You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Antonio Piccolboni (JIRA)" <ji...@apache.org> on 2012/06/29 22:53:44 UTC
[jira] [Commented] (HADOOP-4842) Streaming combiner should allow
command, not just JavaClass
[ https://issues.apache.org/jira/browse/HADOOP-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404206#comment-13404206 ]
Antonio Piccolboni commented on HADOOP-4842:
--------------------------------------------
I entered a comment on HADOOP-1722 that may be of interest here too. The problem seems to be that binary formats and streaming combiners don't work well together particularly if one want the reducer to read typedbytes and write text. If the combiner does the same then we have the combiner write text while the reducer expects typedbytes. Trying to understand what the expected behavior is before I submit a bug.
> Streaming combiner should allow command, not just JavaClass
> -----------------------------------------------------------
>
> Key: HADOOP-4842
> URL: https://issues.apache.org/jira/browse/HADOOP-4842
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Marco Nicosia
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-4842-1.txt, patch-4842-2.txt, patch-4842-3.txt, patch-4842.txt
>
>
> Streaming jobs are way slower than Java jobs for many reasons, but certainly stopping the shell-only programmer from using the combiner feature won't help. Right now, the streaming usage says:
> {quote}
> -mapper <cmd|JavaClassName> The streaming command to run
> -combiner <JavaClassName> Combiner has to be a Java class
> -reducer <cmd|JavaClassName> The streaming command to run
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira