You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2010/05/12 21:28:43 UTC
[jira] Created: (MAPREDUCE-1785) Add streaming config option for
not emitting the key
Add streaming config option for not emitting the key
----------------------------------------------------
Key: MAPREDUCE-1785
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1785
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/streaming
Affects Versions: 0.22.0
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Minor
Fix For: 0.22.0
PipeMapper currently does not emit the key when using TextInputFormat. If you switch to input formats (eg LzoTextInputFormat) the key will be emitted. We should add an option so users can explicitly make streaming not emit the key so they can change input formats without breaking or having to modify their existing programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1785) Add streaming config option for
not emitting the key
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867556#action_12867556 ]
Tom White commented on MAPREDUCE-1785:
--------------------------------------
+1
> Add streaming config option for not emitting the key
> ----------------------------------------------------
>
> Key: MAPREDUCE-1785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1785
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/streaming
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce-1785-1.patch
>
>
> PipeMapper currently does not emit the key when using TextInputFormat. If you switch to input formats (eg LzoTextInputFormat) the key will be emitted. We should add an option so users can explicitly make streaming not emit the key so they can change input formats without breaking or having to modify their existing programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1785) Add streaming config option for
not emitting the key
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated MAPREDUCE-1785:
-----------------------------------
Status: Patch Available (was: Open)
> Add streaming config option for not emitting the key
> ----------------------------------------------------
>
> Key: MAPREDUCE-1785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1785
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/streaming
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce-1785-1.patch
>
>
> PipeMapper currently does not emit the key when using TextInputFormat. If you switch to input formats (eg LzoTextInputFormat) the key will be emitted. We should add an option so users can explicitly make streaming not emit the key so they can change input formats without breaking or having to modify their existing programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1785) Add streaming config option for
not emitting the key
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867668#action_12867668 ]
Hadoop QA commented on MAPREDUCE-1785:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12444445/mapreduce-1785-1.patch
against trunk revision 944082.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/188/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/188/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/188/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/188/console
This message is automatically generated.
> Add streaming config option for not emitting the key
> ----------------------------------------------------
>
> Key: MAPREDUCE-1785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1785
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/streaming
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce-1785-1.patch
>
>
> PipeMapper currently does not emit the key when using TextInputFormat. If you switch to input formats (eg LzoTextInputFormat) the key will be emitted. We should add an option so users can explicitly make streaming not emit the key so they can change input formats without breaking or having to modify their existing programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1785) Add streaming config option for
not emitting the key
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874417#action_12874417 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1785:
----------------------------------------------------
Patch looks good to me also.
bq. -1 core tests.
TestSpecialCharactersInOutputPath failed with NoClassDefFoundError. I ran the same test on my machine with the patch. It ran successfully.
> Add streaming config option for not emitting the key
> ----------------------------------------------------
>
> Key: MAPREDUCE-1785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1785
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/streaming
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce-1785-1.patch
>
>
> PipeMapper currently does not emit the key when using TextInputFormat. If you switch to input formats (eg LzoTextInputFormat) the key will be emitted. We should add an option so users can explicitly make streaming not emit the key so they can change input formats without breaking or having to modify their existing programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1785) Add streaming config option for
not emitting the key
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated MAPREDUCE-1785:
-----------------------------------
Attachment: mapreduce-1785-1.patch
Patch attached.
* Adds stream.map.input.ignoreKey for toggling key emission. The default behavior is unchanged.
* Updated streaming.xml docs and added test coverage in TestStreamingKeyValue
> Add streaming config option for not emitting the key
> ----------------------------------------------------
>
> Key: MAPREDUCE-1785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1785
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/streaming
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce-1785-1.patch
>
>
> PipeMapper currently does not emit the key when using TextInputFormat. If you switch to input formats (eg LzoTextInputFormat) the key will be emitted. We should add an option so users can explicitly make streaming not emit the key so they can change input formats without breaking or having to modify their existing programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1785) Add streaming config option for
not emitting the key
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-1785:
-----------------------------------------------
Release Note: Added a configuration property "stream.map.input.ignoreKey" to specify whether to ignore key or not while writing input for the mapper. This configuration parameter is valid only if stream.map.input.writer.class is org.apache.hadoop.streaming.io.TextInputWriter.class. For all other InputWriter's, key is always written. (was: Added a configuration property "stream.map.input.ignoreKey" to specify whether to ignore key or not while reading input.)
> Add streaming config option for not emitting the key
> ----------------------------------------------------
>
> Key: MAPREDUCE-1785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1785
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/streaming
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce-1785-1.patch
>
>
> PipeMapper currently does not emit the key when using TextInputFormat. If you switch to input formats (eg LzoTextInputFormat) the key will be emitted. We should add an option so users can explicitly make streaming not emit the key so they can change input formats without breaking or having to modify their existing programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1785) Add streaming config option for
not emitting the key
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated MAPREDUCE-1785:
--------------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
I committed this. Thanks Eli.
> Add streaming config option for not emitting the key
> ----------------------------------------------------
>
> Key: MAPREDUCE-1785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1785
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/streaming
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce-1785-1.patch
>
>
> PipeMapper currently does not emit the key when using TextInputFormat. If you switch to input formats (eg LzoTextInputFormat) the key will be emitted. We should add an option so users can explicitly make streaming not emit the key so they can change input formats without breaking or having to modify their existing programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1785) Add streaming config option for
not emitting the key
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-1785:
-----------------------------------------------
Release Note: Added a configuration property "stream.map.input.ignoreKey" to specify whether to ignore key or not while reading input.
> Add streaming config option for not emitting the key
> ----------------------------------------------------
>
> Key: MAPREDUCE-1785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1785
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/streaming
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce-1785-1.patch
>
>
> PipeMapper currently does not emit the key when using TextInputFormat. If you switch to input formats (eg LzoTextInputFormat) the key will be emitted. We should add an option so users can explicitly make streaming not emit the key so they can change input formats without breaking or having to modify their existing programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.