You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Roman Shaposhnik (Created) (JIRA)" <ji...@apache.org> on 2012/02/09 23:52:57 UTC
[jira] [Created] (MAPREDUCE-3845) hadoop distcp fails to run with
java.lang.NoClassDefFoundError
hadoop distcp fails to run with java.lang.NoClassDefFoundError
--------------------------------------------------------------
Key: MAPREDUCE-3845
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3845
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 0.23.1
Reporter: Roman Shaposhnik
Here's how to reproduce:
{noformat}
$ cd /tmnp
$ curl http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/hadoop-0.23.1.tar.gz | tar xzvf -
$ cd hadoop-0.23.1
$ ./bin/hadoop distcp
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCp
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.hadoop.tools.DistCp. Program will exit.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3845) hadoop distcp fails to run with
java.lang.NoClassDefFoundError
Posted by "Roman Shaposhnik (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205162#comment-13205162 ]
Roman Shaposhnik commented on MAPREDUCE-3845:
---------------------------------------------
Yup, sorry about the noise. It is now fixed and hence I'm closing the issue.
Quick question though -- would it make sense to deprecate these 2 commands along the lines with everything else that got moved to mapred. Will the patch along the following lines make sense:
{noformat}
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
@@ -66,7 +66,7 @@ case $COMMAND in
;;
#mapred commands for backwards compatibility
- pipes|job|queue)
+ pipes|job|queue|distcp|archive)
echo "DEPRECATED: Use of this script to execute mapred command is deprecated." 1>&2
echo "Instead use the mapred command for it." 1>&2
echo "" 1>&2
@@ -100,17 +100,9 @@ case $COMMAND in
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
elif [ "$COMMAND" = "jar" ] ; then
CLASS=org.apache.hadoop.util.RunJar
- elif [ "$COMMAND" = "distcp" ] ; then
- CLASS=org.apache.hadoop.tools.DistCp
- CLASSPATH=${CLASSPATH}:${TOOL_PATH}
- HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
elif [ "$COMMAND" = "daemonlog" ] ; then
CLASS=org.apache.hadoop.log.LogLevel
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
- elif [ "$COMMAND" = "archive" ] ; then
- CLASS=org.apache.hadoop.tools.HadoopArchives
- CLASSPATH=${CLASSPATH}:${TOOL_PATH}
- HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
elif [[ "$COMMAND" = -* ]] ; then
# class and package names cannot begin with a -
echo "Error: No command named \`$COMMAND' was found. Perhaps you meant \`hadoop ${COMMAND#-}'"
{noformat}
> hadoop distcp fails to run with java.lang.NoClassDefFoundError
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3845
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3845
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 0.23.1
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Fix For: 0.23.1
>
>
> Here's how to reproduce:
> {noformat}
> $ cd /tmnp
> $ curl http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/hadoop-0.23.1.tar.gz | tar xzvf -
> $ cd hadoop-0.23.1
> $ ./bin/hadoop distcp
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCp
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.hadoop.tools.DistCp. Program will exit.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3845) hadoop distcp fails to run with
java.lang.NoClassDefFoundError
Posted by "Roman Shaposhnik (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated MAPREDUCE-3845:
----------------------------------------
Fix Version/s: 0.23.1
Assignee: Roman Shaposhnik
> hadoop distcp fails to run with java.lang.NoClassDefFoundError
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3845
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3845
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 0.23.1
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Priority: Blocker
> Fix For: 0.23.1
>
>
> Here's how to reproduce:
> {noformat}
> $ cd /tmnp
> $ curl http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/hadoop-0.23.1.tar.gz | tar xzvf -
> $ cd hadoop-0.23.1
> $ ./bin/hadoop distcp
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCp
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.hadoop.tools.DistCp. Program will exit.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3845) hadoop distcp fails to run with
java.lang.NoClassDefFoundError
Posted by "Roman Shaposhnik (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated MAPREDUCE-3845:
----------------------------------------
Priority: Major (was: Blocker)
Apparently distcp command in mapred launcher script works. Bumping down the priority since a simple workaround exists.
> hadoop distcp fails to run with java.lang.NoClassDefFoundError
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3845
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3845
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 0.23.1
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Fix For: 0.23.1
>
>
> Here's how to reproduce:
> {noformat}
> $ cd /tmnp
> $ curl http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/hadoop-0.23.1.tar.gz | tar xzvf -
> $ cd hadoop-0.23.1
> $ ./bin/hadoop distcp
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCp
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.hadoop.tools.DistCp. Program will exit.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3845) hadoop distcp fails to run with
java.lang.NoClassDefFoundError
Posted by "Roman Shaposhnik (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated MAPREDUCE-3845:
----------------------------------------
Priority: Blocker (was: Major)
> hadoop distcp fails to run with java.lang.NoClassDefFoundError
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3845
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3845
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 0.23.1
> Reporter: Roman Shaposhnik
> Priority: Blocker
> Fix For: 0.23.1
>
>
> Here's how to reproduce:
> {noformat}
> $ cd /tmnp
> $ curl http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/hadoop-0.23.1.tar.gz | tar xzvf -
> $ cd hadoop-0.23.1
> $ ./bin/hadoop distcp
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCp
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.hadoop.tools.DistCp. Program will exit.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3845) hadoop distcp fails to run with
java.lang.NoClassDefFoundError
Posted by "Roman Shaposhnik (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik resolved MAPREDUCE-3845.
-----------------------------------------
Resolution: Duplicate
DUP of HADOOP-7999
> hadoop distcp fails to run with java.lang.NoClassDefFoundError
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3845
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3845
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 0.23.1
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Fix For: 0.23.1
>
>
> Here's how to reproduce:
> {noformat}
> $ cd /tmnp
> $ curl http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/hadoop-0.23.1.tar.gz | tar xzvf -
> $ cd hadoop-0.23.1
> $ ./bin/hadoop distcp
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCp
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.hadoop.tools.DistCp. Program will exit.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3845) hadoop distcp fails to run with
java.lang.NoClassDefFoundError
Posted by "Jason Lowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205055#comment-13205055 ]
Jason Lowe commented on MAPREDUCE-3845:
---------------------------------------
This seems related to the same type of problem as in HADOOP-7999 which should be fixed in 0.23. I'm unable to reproduce the problem, as I get the same output from both commands:
{noformat}
$ ls
hadoop-0.23.1.tar.gz
$ tar xf hadoop\-0.23.1.tar.gz
$ cd hadoop\-0.23.1
$ ./bin/hadoop distcp
12/02/09 17:40:23 ERROR tools.DistCp: Invalid arguments:
java.lang.IllegalArgumentException: Target path not specified
at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:86)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:95)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:357)
Invalid arguments: Target path not specified
usage: distcp OPTIONS [source_path...] <target_path>
OPTIONS
-async Should distcp execution be blocking
-atomic Commit all changes or none
-bandwidth <arg> Specify bandwidth per map in MB
-delete Delete from target, files missing in source
-f <arg> List of files that need to be copied
-filelimit <arg> (Deprecated!) Limit number of files copied to <= n
-i Ignore failures during copy
-log <arg> Folder on DFS where distcp execution logs are
saved
-m <arg> Max number of concurrent maps to use for copy
-mapredSslConf <arg> Configuration for ssl config file, to use with
hftps://
-overwrite Choose to overwrite target files unconditionally,
even if they exist.
-p <arg> preserve status (rbugp)(replication, block-size,
user, group, permission)
-sizelimit <arg> (Deprecated!) Limit number of files copied to <= n
bytes
-skipcrccheck Whether to skip CRC checks between source and
target paths.
-strategy <arg> Copy strategy to use. Default is dividing work
based on file sizes
-tmp <arg> Intermediate work path to be used for atomic
commit
-update Update target, copying only missingfiles or
directories
$ ./bin/mapred distcp
12/02/09 17:40:29 ERROR tools.DistCp: Invalid arguments:
java.lang.IllegalArgumentException: Target path not specified
at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:86)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:95)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:357)
Invalid arguments: Target path not specified
usage: distcp OPTIONS [source_path...] <target_path>
OPTIONS
-async Should distcp execution be blocking
-atomic Commit all changes or none
-bandwidth <arg> Specify bandwidth per map in MB
-delete Delete from target, files missing in source
-f <arg> List of files that need to be copied
-filelimit <arg> (Deprecated!) Limit number of files copied to <= n
-i Ignore failures during copy
-log <arg> Folder on DFS where distcp execution logs are
saved
-m <arg> Max number of concurrent maps to use for copy
-mapredSslConf <arg> Configuration for ssl config file, to use with
hftps://
-overwrite Choose to overwrite target files unconditionally,
even if they exist.
-p <arg> preserve status (rbugp)(replication, block-size,
user, group, permission)
-sizelimit <arg> (Deprecated!) Limit number of files copied to <= n
bytes
-skipcrccheck Whether to skip CRC checks between source and
target paths.
-strategy <arg> Copy strategy to use. Default is dividing work
based on file sizes
-tmp <arg> Intermediate work path to be used for atomic
commit
-update Update target, copying only missingfiles or
directories
$
{noformat}
Normally the distcp jar is located via TOOL_PATH which is setup in hadoop-config.sh if not overridden in the environment. In both the {{hadoop}} and {{mapred}} cases I'm seeing TOOL_PATH set via hadoop-config.sh. It'd be interesting to see how one script is picking up the proper TOOL_PATH but the other isn't (e.g.: prepend {{bash -x}} to each shell command and compare the output)
> hadoop distcp fails to run with java.lang.NoClassDefFoundError
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3845
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3845
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 0.23.1
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Fix For: 0.23.1
>
>
> Here's how to reproduce:
> {noformat}
> $ cd /tmnp
> $ curl http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/hadoop-0.23.1.tar.gz | tar xzvf -
> $ cd hadoop-0.23.1
> $ ./bin/hadoop distcp
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCp
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.hadoop.tools.DistCp. Program will exit.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3845) hadoop distcp fails to run with
java.lang.NoClassDefFoundError
Posted by "Tsz Wo (Nicholas), SZE (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205065#comment-13205065 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-3845:
---------------------------------------------------
> I'm unable to reproduce the problem, as I get the same output from both commands ...
I beg you already have fixed it in HADOOP-7999 since both archives and distcp are under hadoop-tools/ .
-----
Roman, does it work for you now?
> hadoop distcp fails to run with java.lang.NoClassDefFoundError
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3845
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3845
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 0.23.1
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Fix For: 0.23.1
>
>
> Here's how to reproduce:
> {noformat}
> $ cd /tmnp
> $ curl http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/hadoop-0.23.1.tar.gz | tar xzvf -
> $ cd hadoop-0.23.1
> $ ./bin/hadoop distcp
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCp
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.hadoop.tools.DistCp. Program will exit.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira