You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Devaraj K (Created) (JIRA)" <ji...@apache.org> on 2012/03/28 15:38:26 UTC
[jira] [Created] (MAPREDUCE-4077) Issue while using Hadoop
Streaming job
Issue while using Hadoop Streaming job
--------------------------------------
Key: MAPREDUCE-4077
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4077
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
When we use -file option it says deprecated and use -files.
{code:xml}
linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop
-output /test/output/3 -mapper cat -reducer wc -file hadoop
02/02/19 10:55:51 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
{code}
But when we use -files option, it says unrecognized option.
{code:xml}
linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
/test/output/3 -mapper cat -reducer wc -files hadoop
02/02/19 10:56:42 ERROR streaming.StreamJob: Unrecognized option: -files
Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
{code}
When we use -archives option, it says unrecognized option.
{code:xml}
linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
/test/output/3 -mapper cat -reducer wc -archives testarchive.rar
02/02/19 11:05:43 ERROR streaming.StreamJob: Unrecognized option: -archives
Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
{code}
But in the options it will display the usage of the -archives.
{code:xml}
linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
/test/output/3 -mapper cat -reducer wc -archives testarchive.rar
02/02/19 11:05:43 ERROR streaming.StreamJob: Unrecognized option: -archives
Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
..........
..........
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4077) Issues while using Hadoop
Streaming job
Posted by "Devaraj K (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240481#comment-13240481 ]
Devaraj K commented on MAPREDUCE-4077:
--------------------------------------
I got the usage of these options, but for an user/beginner without going through the code it will be difficult to understand the usage of these options by seeing the below -info detailed usage
{code:xml}
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
{code}
I suggest to change the above messages like this
{code:xml}
Generic options supported are
-Dconf=<configuration file> specify an application configuration file
-D <property=value> use value for given property
-Dfs=<local|namenode:port> specify a namenode
-Djt=<local|jobtracker:port> specify a job tracker
-Dfiles=<comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-Dlibjars=<comma separated list of jars> specify comma separated jar files to include in the classpath.
-Darchives=<comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
{code}
{code:xml}
02/02/19 10:55:51 WARN streaming.StreamJob: -file option is deprecated, please use generic option -Dfiles instead.
{code}
{code:xml}
02/02/19 10:55:51 WARN streaming.StreamJob: -cacheArchive option is deprecated, please use -Darchives instead.
{code}
> Issues while using Hadoop Streaming job
> ---------------------------------------
>
> Key: MAPREDUCE-4077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Devaraj K
> Assignee: Devaraj K
>
> When we use -file option it says deprecated and use -files.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop
> -output /test/output/3 -mapper cat -reducer wc -file hadoop
> 02/02/19 10:55:51 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
> {code}
> But when we use -files option, it says unrecognized option.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -files hadoop
> 02/02/19 10:56:42 ERROR streaming.StreamJob: Unrecognized option: -files
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> {code}
> When we use -archives option, it says unrecognized option.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -archives testarchive.rar
> 02/02/19 11:05:43 ERROR streaming.StreamJob: Unrecognized option: -archives
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> {code}
> But in the options it will display the usage of the -archives.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -archives testarchive.rar
> 02/02/19 11:05:43 ERROR streaming.StreamJob: Unrecognized option: -archives
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> ..........
> ..........
> -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
> -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4077) Issues while using Hadoop
Streaming job
Posted by "Devaraj K (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K reopened MAPREDUCE-4077:
----------------------------------
Reopening this issue to improve the usage info.
> Issues while using Hadoop Streaming job
> ---------------------------------------
>
> Key: MAPREDUCE-4077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Devaraj K
> Assignee: Devaraj K
>
> When we use -file option it says deprecated and use -files.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop
> -output /test/output/3 -mapper cat -reducer wc -file hadoop
> 02/02/19 10:55:51 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
> {code}
> But when we use -files option, it says unrecognized option.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -files hadoop
> 02/02/19 10:56:42 ERROR streaming.StreamJob: Unrecognized option: -files
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> {code}
> When we use -archives option, it says unrecognized option.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -archives testarchive.rar
> 02/02/19 11:05:43 ERROR streaming.StreamJob: Unrecognized option: -archives
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> {code}
> But in the options it will display the usage of the -archives.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -archives testarchive.rar
> 02/02/19 11:05:43 ERROR streaming.StreamJob: Unrecognized option: -archives
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> ..........
> ..........
> -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
> -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4077) Issues while using Hadoop
Streaming job
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240491#comment-13240491 ]
Robert Joseph Evans commented on MAPREDUCE-4077:
------------------------------------------------
It is not a -D option.
-D conf=file.conf
will not work. It has to be -conf file.conf. But all generic options must appear on the command line before any of the non-generic options like -input, -output, etc.
The Generic Options Supported message is correct, but it does not indicate where those options are allowed on the command line and where they are not allowed. If you want to reopen the ticket to improve the message that would be great.
> Issues while using Hadoop Streaming job
> ---------------------------------------
>
> Key: MAPREDUCE-4077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Devaraj K
> Assignee: Devaraj K
>
> When we use -file option it says deprecated and use -files.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop
> -output /test/output/3 -mapper cat -reducer wc -file hadoop
> 02/02/19 10:55:51 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
> {code}
> But when we use -files option, it says unrecognized option.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -files hadoop
> 02/02/19 10:56:42 ERROR streaming.StreamJob: Unrecognized option: -files
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> {code}
> When we use -archives option, it says unrecognized option.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -archives testarchive.rar
> 02/02/19 11:05:43 ERROR streaming.StreamJob: Unrecognized option: -archives
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> {code}
> But in the options it will display the usage of the -archives.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -archives testarchive.rar
> 02/02/19 11:05:43 ERROR streaming.StreamJob: Unrecognized option: -archives
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> ..........
> ..........
> -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
> -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4077) Issues while using Hadoop
Streaming job
Posted by "Robert Joseph Evans (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans resolved MAPREDUCE-4077.
--------------------------------------------
Resolution: Not A Problem
The issue is with the order of your arguments. I know this is ugly, but it is difficult to fix for various backwards compatibility reasons. The -archives and -files need to come before the -input argument. They need to be at the beginning like with the -D arguments.
> Issues while using Hadoop Streaming job
> ---------------------------------------
>
> Key: MAPREDUCE-4077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Devaraj K
> Assignee: Devaraj K
>
> When we use -file option it says deprecated and use -files.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop
> -output /test/output/3 -mapper cat -reducer wc -file hadoop
> 02/02/19 10:55:51 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
> {code}
> But when we use -files option, it says unrecognized option.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -files hadoop
> 02/02/19 10:56:42 ERROR streaming.StreamJob: Unrecognized option: -files
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> {code}
> When we use -archives option, it says unrecognized option.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -archives testarchive.rar
> 02/02/19 11:05:43 ERROR streaming.StreamJob: Unrecognized option: -archives
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> {code}
> But in the options it will display the usage of the -archives.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -archives testarchive.rar
> 02/02/19 11:05:43 ERROR streaming.StreamJob: Unrecognized option: -archives
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> ..........
> ..........
> -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
> -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4077) Issues while using Hadoop
Streaming job
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-4077:
---------------------------------
Summary: Issues while using Hadoop Streaming job (was: Issue while using Hadoop Streaming job)
> Issues while using Hadoop Streaming job
> ---------------------------------------
>
> Key: MAPREDUCE-4077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Devaraj K
> Assignee: Devaraj K
>
> When we use -file option it says deprecated and use -files.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop
> -output /test/output/3 -mapper cat -reducer wc -file hadoop
> 02/02/19 10:55:51 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
> {code}
> But when we use -files option, it says unrecognized option.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -files hadoop
> 02/02/19 10:56:42 ERROR streaming.StreamJob: Unrecognized option: -files
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> {code}
> When we use -archives option, it says unrecognized option.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -archives testarchive.rar
> 02/02/19 11:05:43 ERROR streaming.StreamJob: Unrecognized option: -archives
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> {code}
> But in the options it will display the usage of the -archives.
> {code:xml}
> linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./hadoop jar
> ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop -output
> /test/output/3 -mapper cat -reducer wc -archives testarchive.rar
> 02/02/19 11:05:43 ERROR streaming.StreamJob: Unrecognized option: -archives
> Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
> ..........
> ..........
> -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
> -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira