You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jason Lowe (Commented) (JIRA)" <ji...@apache.org> on 2012/02/10 00:47:59 UTC
[jira] [Commented] (MAPREDUCE-3845) hadoop distcp fails to run with
java.lang.NoClassDefFoundError
[ https://issues.apache.org/jira/browse/MAPREDUCE-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205055#comment-13205055 ]
Jason Lowe commented on MAPREDUCE-3845:
---------------------------------------
This seems related to the same type of problem as in HADOOP-7999 which should be fixed in 0.23. I'm unable to reproduce the problem, as I get the same output from both commands:
{noformat}
$ ls
hadoop-0.23.1.tar.gz
$ tar xf hadoop\-0.23.1.tar.gz
$ cd hadoop\-0.23.1
$ ./bin/hadoop distcp
12/02/09 17:40:23 ERROR tools.DistCp: Invalid arguments:
java.lang.IllegalArgumentException: Target path not specified
at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:86)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:95)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:357)
Invalid arguments: Target path not specified
usage: distcp OPTIONS [source_path...] <target_path>
OPTIONS
-async Should distcp execution be blocking
-atomic Commit all changes or none
-bandwidth <arg> Specify bandwidth per map in MB
-delete Delete from target, files missing in source
-f <arg> List of files that need to be copied
-filelimit <arg> (Deprecated!) Limit number of files copied to <= n
-i Ignore failures during copy
-log <arg> Folder on DFS where distcp execution logs are
saved
-m <arg> Max number of concurrent maps to use for copy
-mapredSslConf <arg> Configuration for ssl config file, to use with
hftps://
-overwrite Choose to overwrite target files unconditionally,
even if they exist.
-p <arg> preserve status (rbugp)(replication, block-size,
user, group, permission)
-sizelimit <arg> (Deprecated!) Limit number of files copied to <= n
bytes
-skipcrccheck Whether to skip CRC checks between source and
target paths.
-strategy <arg> Copy strategy to use. Default is dividing work
based on file sizes
-tmp <arg> Intermediate work path to be used for atomic
commit
-update Update target, copying only missingfiles or
directories
$ ./bin/mapred distcp
12/02/09 17:40:29 ERROR tools.DistCp: Invalid arguments:
java.lang.IllegalArgumentException: Target path not specified
at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:86)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:95)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:357)
Invalid arguments: Target path not specified
usage: distcp OPTIONS [source_path...] <target_path>
OPTIONS
-async Should distcp execution be blocking
-atomic Commit all changes or none
-bandwidth <arg> Specify bandwidth per map in MB
-delete Delete from target, files missing in source
-f <arg> List of files that need to be copied
-filelimit <arg> (Deprecated!) Limit number of files copied to <= n
-i Ignore failures during copy
-log <arg> Folder on DFS where distcp execution logs are
saved
-m <arg> Max number of concurrent maps to use for copy
-mapredSslConf <arg> Configuration for ssl config file, to use with
hftps://
-overwrite Choose to overwrite target files unconditionally,
even if they exist.
-p <arg> preserve status (rbugp)(replication, block-size,
user, group, permission)
-sizelimit <arg> (Deprecated!) Limit number of files copied to <= n
bytes
-skipcrccheck Whether to skip CRC checks between source and
target paths.
-strategy <arg> Copy strategy to use. Default is dividing work
based on file sizes
-tmp <arg> Intermediate work path to be used for atomic
commit
-update Update target, copying only missingfiles or
directories
$
{noformat}
Normally the distcp jar is located via TOOL_PATH which is setup in hadoop-config.sh if not overridden in the environment. In both the {{hadoop}} and {{mapred}} cases I'm seeing TOOL_PATH set via hadoop-config.sh. It'd be interesting to see how one script is picking up the proper TOOL_PATH but the other isn't (e.g.: prepend {{bash -x}} to each shell command and compare the output)
> hadoop distcp fails to run with java.lang.NoClassDefFoundError
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3845
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3845
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 0.23.1
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Fix For: 0.23.1
>
>
> Here's how to reproduce:
> {noformat}
> $ cd /tmnp
> $ curl http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/hadoop-0.23.1.tar.gz | tar xzvf -
> $ cd hadoop-0.23.1
> $ ./bin/hadoop distcp
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCp
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.hadoop.tools.DistCp. Program will exit.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira