You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Александр Малышевский <ma...@iqmen.ru> on 2012/04/13 13:28:48 UTC

MultiGlobal scheduling in the Fair Scheduler

Hello.

I'm using hadoop 0.20.203.0 and fair scheduler. And I have some difficulties
concerning the lack of multiple tasks per heartbeat in fair scheduler in
version 0.20.203.0. 

I've found fixed feature about it -
https://issues.apache.org/jira/browse/MAPREDUCE-548. Fix version is
<https://issues.apache.org/jira/browse/MAPREDUCE/fixforversion/12314045>
0.21.0. Version 0.20.203.0 was released several months later than 0.21.0,
but as I can see this feature was not included in 0.20.203.0.

I don't want to upgrade my hadoop to another version cause it may make me
change lots of code. So the question is can I upgrade only fair scheduler
jar without upgrading other libraries? I just need multiple tasks per
heartbeat in fair scheduler. What's the easiest way to get it? I tried to
upgrade fair scheduler jar to 1.0.2 but it didn't worked with the following
exception:

 

2012-04-12 17:33:42,001 FATAL org.apache.hadoop.mapred.JobTracker:
java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/TaskType

       at
org.apache.hadoop.mapred.FairScheduler.<clinit>(FairScheduler.java:64)

       at java.lang.Class.forName0(Native Method)

       at java.lang.Class.forName(Class.java:247)

       at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)

       at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)

       at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)

       at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1573)

       at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)

       at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)

       at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)

Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapreduce.TaskType

       at java.net.URLClassLoader$1.run(URLClassLoader.java:202)

       at java.security.AccessController.doPrivileged(Native Method)

       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)

       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

       ... 10 more

 

2012-04-12 17:33:42,002 INFO org.apache.hadoop.mapred.JobTracker:
SHUTDOWN_MSG: