You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Albert Chern (JIRA)" <ji...@apache.org> on 2007/01/25 00:22:49 UTC

[jira] Created: (HADOOP-927) MapReduce is Broken for User-Defined Classes

MapReduce is Broken for User-Defined Classes
--------------------------------------------

                 Key: HADOOP-927
                 URL: https://issues.apache.org/jira/browse/HADOOP-927
             Project: Hadoop
          Issue Type: Bug
          Components: mapred
    Affects Versions: 0.11.0
            Reporter: Albert Chern
            Priority: Critical


Checked out trunk today and all of my jobs with a user-defined class as a map output value class fail with a ClassNotFoundException.  Here's the stack trace:

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: test.StringWrapper
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
	at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:406)
	at org.apache.hadoop.mapred.ReduceTaskRunner.(ReduceTaskRunner.java:339)
	at org.apache.hadoop.mapred.ReduceTask.createRunner(ReduceTask.java:91)
	at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:983)
	at org.apache.hadoop.mapred.TaskTracker.launchTaskForJob(TaskTracker.java:366)
	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:359)
	at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:823)
	at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:510)
	at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:852)
	at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1494)

I haven't looked deeply into it, but I would guess that the problem is in the constructor of ReduceTaskRunner where it attempts to create a SequenceFile.Sorter for the map output value class.  It's probably done in the main TaskTracker JVM process, so external libraries are not on its classpath.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-927) MapReduce is Broken for User-Defined Classes

Posted by "liupengfei (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12510078 ] 

liupengfei commented on HADOOP-927:
-----------------------------------

This issue is still exist in hadoop-0.13.0:
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.qnsoft.asm.SMS
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:351)
	at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:407)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:308)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:123)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1707)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.qnsoft.asm.SMS
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:331)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:343)
	... 4 more
Caused by: java.lang.ClassNotFoundException: com.qnsoft.asm.SMS
	at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:242)
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:318)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:329)
	... 5 more

I have defined SMS extends Writable, it just cannot run under hadoop-0.13.0,but ok in 0.9.2.

> MapReduce is Broken for User-Defined Classes
> --------------------------------------------
>
>                 Key: HADOOP-927
>                 URL: https://issues.apache.org/jira/browse/HADOOP-927
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.11.0
>            Reporter: Albert Chern
>            Priority: Critical
>
> Checked out trunk today and all of my jobs with a user-defined class as a map output value class fail with a ClassNotFoundException.  Here's the stack trace:
> java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: test.StringWrapper
> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
> 	at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:406)
> 	at org.apache.hadoop.mapred.ReduceTaskRunner.(ReduceTaskRunner.java:339)
> 	at org.apache.hadoop.mapred.ReduceTask.createRunner(ReduceTask.java:91)
> 	at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:983)
> 	at org.apache.hadoop.mapred.TaskTracker.launchTaskForJob(TaskTracker.java:366)
> 	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:359)
> 	at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:823)
> 	at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:510)
> 	at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:852)
> 	at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1494)
> I haven't looked deeply into it, but I would guess that the problem is in the constructor of ReduceTaskRunner where it attempts to create a SequenceFile.Sorter for the map output value class.  It's probably done in the main TaskTracker JVM process, so external libraries are not on its classpath.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (HADOOP-927) MapReduce is Broken for User-Defined Classes

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting resolved HADOOP-927.
---------------------------------

    Resolution: Duplicate

> MapReduce is Broken for User-Defined Classes
> --------------------------------------------
>
>                 Key: HADOOP-927
>                 URL: https://issues.apache.org/jira/browse/HADOOP-927
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.11.0
>            Reporter: Albert Chern
>            Priority: Critical
>
> Checked out trunk today and all of my jobs with a user-defined class as a map output value class fail with a ClassNotFoundException.  Here's the stack trace:
> java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: test.StringWrapper
> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
> 	at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:406)
> 	at org.apache.hadoop.mapred.ReduceTaskRunner.(ReduceTaskRunner.java:339)
> 	at org.apache.hadoop.mapred.ReduceTask.createRunner(ReduceTask.java:91)
> 	at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:983)
> 	at org.apache.hadoop.mapred.TaskTracker.launchTaskForJob(TaskTracker.java:366)
> 	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:359)
> 	at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:823)
> 	at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:510)
> 	at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:852)
> 	at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1494)
> I haven't looked deeply into it, but I would guess that the problem is in the constructor of ReduceTaskRunner where it attempts to create a SequenceFile.Sorter for the map output value class.  It's probably done in the main TaskTracker JVM process, so external libraries are not on its classpath.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-927) MapReduce is Broken for User-Defined Classes

Posted by "Albert Chern (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473524 ] 

Albert Chern commented on HADOOP-927:
-------------------------------------

This issue was the same as Hadoop-964 so it can be closed now.

> MapReduce is Broken for User-Defined Classes
> --------------------------------------------
>
>                 Key: HADOOP-927
>                 URL: https://issues.apache.org/jira/browse/HADOOP-927
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.11.0
>            Reporter: Albert Chern
>            Priority: Critical
>
> Checked out trunk today and all of my jobs with a user-defined class as a map output value class fail with a ClassNotFoundException.  Here's the stack trace:
> java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: test.StringWrapper
> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
> 	at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:406)
> 	at org.apache.hadoop.mapred.ReduceTaskRunner.(ReduceTaskRunner.java:339)
> 	at org.apache.hadoop.mapred.ReduceTask.createRunner(ReduceTask.java:91)
> 	at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:983)
> 	at org.apache.hadoop.mapred.TaskTracker.launchTaskForJob(TaskTracker.java:366)
> 	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:359)
> 	at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:823)
> 	at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:510)
> 	at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:852)
> 	at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1494)
> I haven't looked deeply into it, but I would guess that the problem is in the constructor of ReduceTaskRunner where it attempts to create a SequenceFile.Sorter for the map output value class.  It's probably done in the main TaskTracker JVM process, so external libraries are not on its classpath.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.