You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Aaron Kimball (JIRA)" <ji...@apache.org> on 2009/12/09 01:37:18 UTC
[jira] Created: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
String-to-String Maps should be embeddable in Configuration
-----------------------------------------------------------
Key: HADOOP-6420
URL: https://issues.apache.org/jira/browse/HADOOP-6420
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Attachment: HADOOP-6420.patch
Implementation and test case.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Patch Available (was: Open)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Patch Available (was: Open)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Open (was: Patch Available)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Open (was: Patch Available)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789092#action_12789092 ]
Arun C Murthy commented on HADOOP-6420:
---------------------------------------
Aaron, can you please explain the use case a little more? Why do we really need maps? Why isn't the requisite metadata serializable?
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789163#action_12789163 ]
Aaron Kimball commented on HADOOP-6420:
---------------------------------------
This is used for MAPREDUCE-1126. In HADOOP-6165, the serialization system was changed such that serialization metadata is defined as a {{Map<String, String>}} with an arbitrary set of (key, value) pairs inside.
To support more serialization formats than just WritableSerialization, we need to allow these arbitrary maps of data to be specified by the user via the job's {{Configuration}}. Thus, much as how we have defined mechanisms to embed ints, booleans, and lists of strings in the Configuration, we need to expand the set of datatypes we can embed in the configuration to also include maps of string->string. This does so without needing to worry about escape characters or other clumsiness that would result via embedding a single (k, v) pair in the Configuration.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791139#action_12791139 ]
Hadoop QA commented on HADOOP-6420:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428116/HADOOP-6420.8.patch
against trunk revision 890964.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/208/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/208/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/208/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/208/console
This message is automatically generated.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797357#action_12797357 ]
Doug Cutting commented on HADOOP-6420:
--------------------------------------
I wonder if the map implementation might be considerably smaller if you extend AbstractMap, so that only entrySet() need explicitly throw UnsupportedOperationException?
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790505#action_12790505 ]
Doug Cutting commented on HADOOP-6420:
--------------------------------------
> If iteration isn't required, then this approach sounds good to me.
Great. I think iteration is not required, so let's pursue this.
> After seeing MAPREDUCE-1183, I'm hoping that fewer components will rely on this class.
I don't understand what you're saying here. What do you mean by "this class"? Configuration?
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791124#action_12791124 ]
Aaron Kimball commented on HADOOP-6420:
---------------------------------------
To be more specific; I don't need to know whether the map is _empty_ per se; it is enough to know whether it is _defined_ or not.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Attachment: HADOOP-6420.2.patch
Good points in the review. getMap() now handles variable substitution as well as deprecation.
Since getMap() itself takes the "key" to resolve, I have implemented deprecation-resolution at this level. e.g., "prefix.foo" and "prefix.bar" do not need to be individually deprecated in favor of "newprefix.foo", "newprefix.bar"; instead, you just deprecate "prefix" for "newprefix" as if it were a monolithic entity in the configuration, and calls to {{getMap("prefix")}} will be resolved to look up {{getMap("newprefix")}} instead.
If "newprefix.foo" is further deprecated in favor of "some.other.key", this is *not* resolved by {{getMap("newprefix")}} or {{getMap("prefix")}}.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Attachment: HADOOP-6420.7.patch
Fix javadoc warning.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Patch Available (was: Open)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788308#action_12788308 ]
Hadoop QA commented on HADOOP-6420:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427516/HADOOP-6420.2.patch
against trunk revision 888697.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/181/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/181/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/181/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/181/console
This message is automatically generated.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-6420:
----------------------------------
Status: Open (was: Patch Available)
bq. This does so without needing to worry about escape characters or other clumsiness that would result via embedding a single (k, v) pair in the Configuration.
On the other hand, the current approach ignores inadvertent collisions and {{getMap}} is inefficient and not threadsafe.
bq. much as how we have defined mechanisms to embed ints, booleans, and lists of strings in the Configuration, we need to expand the set of datatypes we can embed in the configuration to also include maps of string\->string
This isn't embedding maps of string\->string; it's selecting a subset of keys in a string\->string map using an unenforced naming convention. It's par for (ab)uses of Configuration, but it's hackish.
Have you looked into using a Stringifier or something along those lines? Escaping shouldn't be too clumsy...
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Open (was: Patch Available)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791170#action_12791170 ]
Chris Douglas commented on HADOOP-6420:
---------------------------------------
bq. What do you mean by "this class"? Configuration?
Yes.
bq. we also support users creating maps using the underlying properties (e.g., conf.set("foo.bar", "a"); conf.set("foo.baz", "b"); conf.getMap("foo")), I've added a declareMap() method that adds the sentinel as well.
This is incoherent. Either the user can set and retrieve string\->string maps from Configuration *or* this is offering an API to Configuration to address a subset of its members behind a preferred interface (i.e. Map<String,String>). In the latter case (_A_), a Map<String,String> wrapping a Configuration instance admits the ad-hoc "map" creation outlined above. In the former case (_B_), this should attempt to guarantee that the Map goes in and out of Configuration without being interfered with, excluding this use case.
Given how general Configuration is, I don't see a way to enforce _B_ invariants without having a blob as the value. If _A_ is the goal- because one wants to add to this sub-map at different stages efficiently, add values via the wrapper interface, or for some other reason- then _by definition_ it creates a sub-map by being interpreted as one, not by declaration.
The implementation details of MAPREDUCE-1126- particularly those enforcing backwards-compatibility- should not bleed into a general Configuration feature. If the metadata API needs to set a sentinel value in the config, then it should set it.
On v8:
# Two {{ConfigItemMap}}s are only equal if they come from the same Configuration; sharing a prefix is insufficient
# What distinguishes {{lookup}} from {{Configuration.this.get(prefix + subKey)}}?
# {{get}}/{{put}} should check for null, to avoid setting/seeking "prefix.null"
# Any particular reason {{putAll}} is unsupported?
# {{ConfigItemMap}} can be a stand-alone or static inner class in conf if (2) is sufficient
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791637#action_12791637 ]
Hadoop QA commented on HADOOP-6420:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428225/HADOOP-6420.9.patch
against trunk revision 891132.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/211/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/211/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/211/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/211/console
This message is automatically generated.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787886#action_12787886 ]
Hadoop QA commented on HADOOP-6420:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427399/HADOOP-6420.patch
against trunk revision 888565.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/180/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/180/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/180/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/180/console
This message is automatically generated.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788972#action_12788972 ]
Hadoop QA commented on HADOOP-6420:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427641/HADOOP-6420.4.patch
against trunk revision 889378.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/191/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/191/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/191/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/191/console
This message is automatically generated.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Patch Available (was: Open)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.10.patch, HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791109#action_12791109 ]
Hadoop QA commented on HADOOP-6420:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428107/HADOOP-6420.7.patch
against trunk revision 890964.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/207/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/207/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/207/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/207/console
This message is automatically generated.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Attachment: HADOOP-6420.4.patch
patch with --no-prefix
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Attachment: HADOOP-6420.3.patch
new patch; sync'd with trunk
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797385#action_12797385 ]
Doug Cutting commented on HADOOP-6420:
--------------------------------------
+1 This looks good to me. Unless there are objections I'll commit this tomorrow.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.10.patch, HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797979#action_12797979 ]
Hudson commented on HADOOP-6420:
--------------------------------
Integrated in Hadoop-Common-trunk #212 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/212/])
. Add functionality permitting subsets of Configuration to be
interpreted as Map<String,String>. Contributed by Aaron Kimball
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: HADOOP-6420.10.patch, HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791053#action_12791053 ]
Hadoop QA commented on HADOOP-6420:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428101/HADOOP-6420.6.patch
against trunk revision 890964.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/206/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/206/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/206/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/206/console
This message is automatically generated.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Open (was: Patch Available)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-6420:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.22.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this. Thanks, Aaron!
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: HADOOP-6420.10.patch, HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795655#action_12795655 ]
Aaron Kimball commented on HADOOP-6420:
---------------------------------------
Chris,
Is this ready to go now?
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Open (was: Patch Available)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788947#action_12788947 ]
Hadoop QA commented on HADOOP-6420:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427639/HADOOP-6420.3.patch
against trunk revision 889378.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/25/console
This message is automatically generated.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Patch Available (was: Open)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Attachment: HADOOP-6420.5.patch
Attaching a patch that uses a view-based Map implementation that doesn't support iteration. I looked through the serializer package and didn't see any uses if size(), entrySet(), or keySet(), so I think this should suffice.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Attachment: HADOOP-6420.10.patch
Good catch. new patch with {{extends AbstractMap}}
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.10.patch, HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Patch Available (was: Open)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796968#action_12796968 ]
Jeff Hammerbacher commented on HADOOP-6420:
-------------------------------------------
Hey Chris,
Any chance you can take a look at this patch? It's blocking 1126, and there has been a patch available for nearly three weeks.
Thanks,
Jeff
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Patch Available (was: Open)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788368#action_12788368 ]
Tom White commented on HADOOP-6420:
-----------------------------------
bq. Since getMap() itself takes the "key" to resolve, I have implemented deprecation-resolution at this level.
+1 This sounds like the right approach to me. Thanks for writing the extra test.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797745#action_12797745 ]
Hudson commented on HADOOP-6420:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #137 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/137/])
. Add functionality permitting subsets of Configuration to be
interpreted as Map<String,String>. Contributed by Aaron Kimball
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: HADOOP-6420.10.patch, HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Patch Available (was: Open)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787917#action_12787917 ]
Tom White commented on HADOOP-6420:
-----------------------------------
A couple of comments on the patch:
* In the documentation for setMap() say that "name" should not have a trailing period.
* Deprecation needs to be handled for getMap(). Also variable substitution. This could be fixed by calling Configuration's get(), rather than the underlying Properties. (You could use the Properties object to get the keys still.) It would be good to add unit tests for these cases.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791573#action_12791573 ]
Aaron Kimball commented on HADOOP-6420:
---------------------------------------
Fair point. I think that the semantics of (A) are clearer. I will modify MAPREDUCE-1126 to use external sentinel values.
As for your other v8 issues
1) Good call on {{equals()}}
2) {{Configuration.this.get()}} will use deprecation-resolution on the provided key. Deprecation-resolution is handled at the map key (prefix) level, not on individual subkeys within the map. So those should not call {{get()}}.
3) Yes.
4) This can be added.
5) I don't see what you mean. If {{Configuration.this.get()}} is used or the current {{Configuration.this.getProps()}}, you still need a {{Configuration.this}} associated with it, so you can't use a static inner class. You could conceivably hold an explicit reference to the {{Configuration}} object, but given that the implicit reference is the whole point of a non-static inner class, I don't see what using a static class buys us.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Attachment: HADOOP-6420.8.patch
I realized after working more on MAPREDUCE-1126 that I need the ability to determine if a map is empty or not; when I was using the HashMap-based (deep-copy) getMap(), this was accomplished via {{Map.size() == 0}}; but determining the map's size in general requires iteration, which we decided not to support due to the expense.
Instead I've added a sentinel value that indicates whether the map exists or not. This is set by {{setMap()}} and can be read with {{mapExists()}}. Since we also support users creating maps using the underlying properties (e.g., {{conf.set("foo.bar", "a"); conf.set("foo.baz", "b"); conf.getMap("foo")}}), I've added a {{declareMap()}} method that adds the sentinel as well. (so you could write: {{conf.set("foo.bar", "a"); conf.set("foo.baz", "b"); conf.declareMap("foo"); assertTrue(conf.mapExists("foo")}}). This works by creating a key named {{"foo"}} in the Configuration whose value is {{"$MAP$"}}.
This is necessary in MAPREDUCE-1126 to determine whether the user has properly called a metadata-API {{setMapOutputKeyClass() / setMapOutputKeySchema()}} method, or whether we should fall back to the deprecated {{JobConf.getMapOutputKeyClass() / JobConf.getOutputKeyClass()}} API.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Attachment: HADOOP-6420.9.patch
Patch #9 addresses code review concerns
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-6420:
----------------------------------
Status: Open (was: Patch Available)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790449#action_12790449 ]
Chris Douglas commented on HADOOP-6420:
---------------------------------------
bq. That could be fun to boostrap. DefaultStringifier uses SerializationFactory which could use DefaultStringifier to find the serializer's metadata. [...]
This is not what I'm talking about. The Stringifier interface obviously doesn't require something as general as DefaultStringifier and nobody has argued that a Map<String,String> is wrong. The issue is whether the current patch offers the best approach to render the metadata to the serialization; its superiority to big, blobby stuff is not obvious. By noting that efforts to push non-primitive types into Configuration has a precedent and an interface, I did not intend to advocate for a pluggable meta-serialization layer on top of Configuration.
bq. Note that we don't need to filter the entire configuration unless someone iterates over the subset. [...]
Returning a Map from Configuration and restricting its range is cleaner and easier to explain. If iteration isn't required, then this approach sounds good to me.
bq. If we did need to support iteration, then perhaps we could change Configuration to use treemap, but that might open other cans of worms.
Agreed. After seeing MAPREDUCE-1183, I'm hoping that fewer components will rely on this class.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Patch Available (was: Open)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790370#action_12790370 ]
Doug Cutting commented on HADOOP-6420:
--------------------------------------
> Have you looked into using a Stringifier or something along those lines?
That could be fun to boostrap. DefaultStringifier uses SerializationFactory which could use DefaultStringifier to find the serializer's metadata.
Whatever we use here we want to be easy for non-Java applications to use. The original reason for using a Map<String,String> model for configuration was that it is a proven format for inter-operable configuration (environment variables). So, if we used something like Stringifier, we'd need to convince ourselves that the serialization we used for this was available to all applications that might want to configure map-valued properties like serialization parameters.
I don't much like filtering, but I also don't much like big blobby stuff in configs.
Note that we don't need to filter the entire configuration unless someone iterates over the subset. The returned map view could instead be a wrapper map implementation that dynamically prefixes keys on access to the underlying configuration. If iteration is not required then we could simply have it throw UnsupportedOperationException. This would address the efficiency and thread-safety concerns.
If we did need to support iteration, then perhaps we could change Configuration to use treemap, but that might open other cans of worms.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797397#action_12797397 ]
Chris Douglas commented on HADOOP-6420:
---------------------------------------
bq. You could conceivably hold an explicit reference to the Configuration object, but given that the implicit reference is the whole point of a non-static inner class, I don't see what using a static class buys us.
I intended (5) to depend on (2); since the deprecation semantics push for an inner class, the point is moot. The static/inner class is just a matter of taste; if the inner class didn't require access to private members, limiting its visibility to public methods keeps the APIs clean.
Given its semantics, {{addMap}} (implying _A_) may be more apt than {{setMap}} (implying _B_), but it's debatable and the documentation makes its semantics clear.
+1
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.10.patch, HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790547#action_12790547 ]
Hadoop QA commented on HADOOP-6420:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427994/HADOOP-6420.5.patch
against trunk revision 890588.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/27/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/27/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/27/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/27/console
This message is automatically generated.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797373#action_12797373 ]
Hadoop QA commented on HADOOP-6420:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12429588/HADOOP-6420.10.patch
against trunk revision 896691.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/261/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/261/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/261/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/261/console
This message is automatically generated.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.10.patch, HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Open (was: Patch Available)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Attachment: HADOOP-6420.6.patch
Good suggestion Steve. Done.
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790662#action_12790662 ]
Steve Loughran commented on HADOOP-6420:
----------------------------------------
Might be handy to make the ConfigItemMap class protected, so that those people who have subclassed a Configuration can work with them
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Patch Available (was: Open)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.2.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6420) String-to-String Maps should be
embeddable in Configuration
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-6420:
----------------------------------
Status: Open (was: Patch Available)
> String-to-String Maps should be embeddable in Configuration
> -----------------------------------------------------------
>
> Key: HADOOP-6420
> URL: https://issues.apache.org/jira/browse/HADOOP-6420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-6420.10.patch, HADOOP-6420.2.patch, HADOOP-6420.3.patch, HADOOP-6420.4.patch, HADOOP-6420.5.patch, HADOOP-6420.6.patch, HADOOP-6420.7.patch, HADOOP-6420.8.patch, HADOOP-6420.9.patch, HADOOP-6420.patch
>
>
> Per MAPREDUCE-1126, we need to be able to take a map of (key, value) pairs and embed that inside a Configuration object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.