You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Peng Lei (JIRA)" <ji...@apache.org> on 2012/12/05 03:40:59 UTC
[jira] [Created] (MAPREDUCE-4847) Command Parsing in Hadoop
Streaming
Peng Lei created MAPREDUCE-4847:
-----------------------------------
Summary: Command Parsing in Hadoop Streaming
Key: MAPREDUCE-4847
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4847
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/streaming
Reporter: Peng Lei
Hadoop streaming parse the mapper and reducer commands by itself, this is not a good choice, when I write a complex mapper/reducer script inline, such as 'perl -ne ...', it don't work.
An alternative way is to send the command to the shell, simply create new process(sh -c "command_and_args"), this not also simplize the streaming code, but also improve its capability!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4847) Command Parsing in Hadoop
Streaming
Posted by "Peng Lei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511031#comment-13511031 ]
Peng Lei commented on MAPREDUCE-4847:
-------------------------------------
Thank you for your comment!
I have put the command in a script file as a workaround, it works. But in this case, the command is not too complex to write a dedicate script file, and on fly script generating is a bit tricky(at least for maintainer).
It seems hadoop can't run on windows without cygwin. Another solution may be: add a new option to instruct streaming to use an alternative command invoker, such as:
-command_invoker "sh -c"
This could solve the issue and didn't break the existing hadoop-streaming application.
-Peng
> Command Parsing in Hadoop Streaming
> -----------------------------------
>
> Key: MAPREDUCE-4847
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4847
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/streaming
> Reporter: Peng Lei
> Labels: features
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Hadoop streaming parse the mapper and reducer commands by itself, this is not a good choice, when I write a complex mapper/reducer script inline, such as 'perl -ne ...', it don't work.
> An alternative way is to send the command to the shell, simply create new process(sh -c "command_and_args"), this not also simplize the streaming code, but also improve its capability!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4847) Command Parsing in Hadoop
Streaming
Posted by "Mariappan Asokan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510236#comment-13510236 ]
Mariappan Asokan commented on MAPREDUCE-4847:
---------------------------------------------
Hi Peng Lei,
By invoking "sh" in Hadoop streaming code, you may be creating a dependency on UNIX(or some ugly branching based on OS.) On Windows, there is no "sh" unless you install additional software like cygwin. Why can't you put your complex command in a file, give execute permission to the file, and then specify the file name as mapper/reducer command? You can create this file on the fly if you want. This way, the streaming command will work in its current form on UNIX and Windows and there is no change needed.
-- Asokan
> Command Parsing in Hadoop Streaming
> -----------------------------------
>
> Key: MAPREDUCE-4847
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4847
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/streaming
> Reporter: Peng Lei
> Labels: features
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Hadoop streaming parse the mapper and reducer commands by itself, this is not a good choice, when I write a complex mapper/reducer script inline, such as 'perl -ne ...', it don't work.
> An alternative way is to send the command to the shell, simply create new process(sh -c "command_and_args"), this not also simplize the streaming code, but also improve its capability!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira