You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Hamedreza Berenjian <ha...@yahoo.com> on 2013/07/20 08:23:59 UTC
a question
Hi,
I have a jar file from a hadoop scheduler(AdaptiveScheduler=resource aware slotless adaptive scheduler).I'm sure that this jar file works properly.I set the required properties in mapred-site.xml correctly, and I put the adaptivescheduler.jar file in the library of hadoop.but when I start my cluster,(using this command: bin/start-all.sh),jobtracker doesn't run.the error is:
2013-07-17 06:07:18,631 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = master/192.168.0.112
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.203.0
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
************************************************************/
2013-07-17 06:07:19,349 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-07-17 06:07:19,361 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2013-07-17 06:07:19,362 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-07-17 06:07:19,362 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics system started
2013-07-17 06:07:20,585 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2013-07-17 06:07:20,588 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-07-17 06:07:20,590 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2013-07-17 06:07:20,632 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2013-07-17 06:07:20,633 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-07-17 06:07:20,665 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2013-07-17 06:07:20,665 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2013-07-17 06:07:20,997 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as maedeh
2013-07-17 06:07:21,050 FATAL org.apache.hadoop.mapred.JobTracker: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.mapred.AdaptiveScheduler not org.apache.hadoop.mapred.TaskScheduler
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2276)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4956)
Caused by: java.lang.RuntimeException: class org.apache.hadoop.mapred.AdaptiveScheduler not org.apache.hadoop.mapred.TaskScheduler
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:892)
... 5 more
2013-07-17 06:07:21,134 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at master/192.168.0.112
************************************************************/
how I can resolve this problem?please help me... my jar file is in this address: /home/maedeh/hadoop-0.20.203.0/lib/adaptivescheduler.jar
Re: a question
Posted by Ted Yu <yu...@gmail.com>.
Adding back common-dev@hadoop.apache.org
Which hadoop version did you use ?
Did you build the jar file against the same hadoop version ?
One possibility was that AdaptiveScheduler from
MAPREDUCE-1380<https://issues.apache.org/jira/browse/MAPREDUCE-1380>doesn't
match the signature of TaskScheduler
in the hadoop release you used.
Cheers
On Sat, Jul 20, 2013 at 7:09 AM, Hamedreza Berenjian <
hamedreza_berenjian@yahoo.com> wrote:
> Hi,
> Yes,exactly. The same scheduler which is in this address:
> https://issues.apache.org/jira/browse/MAPREDUCE-1380
> I attached the jar file. how I can resolve this problem?please help me...
>
> *From:* Ted Yu <yu...@gmail.com>
> *To:* common-dev@hadoop.apache.org; Hamedreza Berenjian <
> hamedreza_berenjian@yahoo.com>
> *Sent:* Saturday, 20 July 2013, 6:52:16
> *Subject:* Re: a question
>
> Did the AdaptiveScheduler come from MAPREDUCE-1380 ?
>
> Thanks
>
> On Fri, Jul 19, 2013 at 11:23 PM, Hamedreza Berenjian <
> hamedreza_berenjian@yahoo.com> wrote:
>
> Hi,
> I have a jar file from a hadoop scheduler(AdaptiveScheduler=resource aware
> slotless adaptive scheduler).I'm sure that this jar file works properly.I
> set the required properties in mapred-site.xml correctly, and I put the
> adaptivescheduler.jar file in the library of hadoop.but when I start my
> cluster,(using this command: bin/start-all.sh),jobtracker doesn't run.the
> error is:
>
> 2013-07-17 06:07:18,631 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = master/192.168.0.112
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.203.0
> STARTUP_MSG: build =
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203-r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
> ************************************************************/
> 2013-07-17 06:07:19,349 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-07-17 06:07:19,361 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-07-17 06:07:19,362 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-07-17 06:07:19,362 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics
> system started
> 2013-07-17 06:07:20,585 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-07-17 06:07:20,588 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-07-17 06:07:20,590 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Updating the current master key for generating delegation tokens
> 2013-07-17 06:07:20,632 INFO org.apache.hadoop.mapred.JobTracker:
> Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2013-07-17 06:07:20,633 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-07-17 06:07:20,665 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Starting expired delegation token remover thread,
> tokenRemoverScanInterval=60 min(s)
> 2013-07-17 06:07:20,665 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Updating the current master key for generating delegation tokens
> 2013-07-17 06:07:20,997 INFO org.apache.hadoop.mapred.JobTracker: Starting
> jobtracker with owner as maedeh
> 2013-07-17 06:07:21,050 FATAL org.apache.hadoop.mapred.JobTracker:
> java.lang.RuntimeException: java.lang.RuntimeException: class
> org.apache.hadoop.mapred.AdaptiveScheduler not
> org.apache.hadoop.mapred.TaskScheduler
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2276)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4956)
> Caused by: java.lang.RuntimeException: class
> org.apache.hadoop.mapred.AdaptiveScheduler not
> org.apache.hadoop.mapred.TaskScheduler
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:892)
> ... 5 more
>
> 2013-07-17 06:07:21,134 INFO org.apache.hadoop.mapred.JobTracker:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down JobTracker at master/192.168.0.112
> ************************************************************/
>
> how I can resolve this problem?please help me... my jar file is in this
> address: /home/maedeh/hadoop-0.20.203.0/lib/adaptivescheduler.jar
>
>
>
>
>
Re: a question
Posted by Ted Yu <yu...@gmail.com>.
Did the AdaptiveScheduler come from MAPREDUCE-1380 ?
Thanks
On Fri, Jul 19, 2013 at 11:23 PM, Hamedreza Berenjian <
hamedreza_berenjian@yahoo.com> wrote:
> Hi,
> I have a jar file from a hadoop scheduler(AdaptiveScheduler=resource aware
> slotless adaptive scheduler).I'm sure that this jar file works properly.I
> set the required properties in mapred-site.xml correctly, and I put the
> adaptivescheduler.jar file in the library of hadoop.but when I start my
> cluster,(using this command: bin/start-all.sh),jobtracker doesn't run.the
> error is:
>
> 2013-07-17 06:07:18,631 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = master/192.168.0.112
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.203.0
> STARTUP_MSG: build =
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203-r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
> ************************************************************/
> 2013-07-17 06:07:19,349 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-07-17 06:07:19,361 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-07-17 06:07:19,362 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-07-17 06:07:19,362 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics
> system started
> 2013-07-17 06:07:20,585 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-07-17 06:07:20,588 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-07-17 06:07:20,590 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Updating the current master key for generating delegation tokens
> 2013-07-17 06:07:20,632 INFO org.apache.hadoop.mapred.JobTracker:
> Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2013-07-17 06:07:20,633 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-07-17 06:07:20,665 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Starting expired delegation token remover thread,
> tokenRemoverScanInterval=60 min(s)
> 2013-07-17 06:07:20,665 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Updating the current master key for generating delegation tokens
> 2013-07-17 06:07:20,997 INFO org.apache.hadoop.mapred.JobTracker: Starting
> jobtracker with owner as maedeh
> 2013-07-17 06:07:21,050 FATAL org.apache.hadoop.mapred.JobTracker:
> java.lang.RuntimeException: java.lang.RuntimeException: class
> org.apache.hadoop.mapred.AdaptiveScheduler not
> org.apache.hadoop.mapred.TaskScheduler
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2276)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4956)
> Caused by: java.lang.RuntimeException: class
> org.apache.hadoop.mapred.AdaptiveScheduler not
> org.apache.hadoop.mapred.TaskScheduler
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:892)
> ... 5 more
>
> 2013-07-17 06:07:21,134 INFO org.apache.hadoop.mapred.JobTracker:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down JobTracker at master/192.168.0.112
> ************************************************************/
>
> how I can resolve this problem?please help me... my jar file is in this
> address: /home/maedeh/hadoop-0.20.203.0/lib/adaptivescheduler.jar
Re: a question
Posted by Shekhar Sharma <sh...@gmail.com>.
Send the constructor of AdaptiveScheduler...
Check the constructor and the "start()" method of your scheduler...
In the start() there is something you are missing..
Regards,
Som Shekhar Sharma
+91-8197243810
On Sat, Jul 20, 2013 at 11:53 AM, Hamedreza Berenjian <
hamedreza_berenjian@yahoo.com> wrote:
> Hi,
> I have a jar file from a hadoop scheduler(AdaptiveScheduler=resource aware
> slotless adaptive scheduler).I'm sure that this jar file works properly.I
> set the required properties in mapred-site.xml correctly, and I put the
> adaptivescheduler.jar file in the library of hadoop.but when I start my
> cluster,(using this command: bin/start-all.sh),jobtracker doesn't run.the
> error is:
>
> 2013-07-17 06:07:18,631 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = master/192.168.0.112
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.203.0
> STARTUP_MSG: build =
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203-r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
> ************************************************************/
> 2013-07-17 06:07:19,349 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-07-17 06:07:19,361 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-07-17 06:07:19,362 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-07-17 06:07:19,362 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics
> system started
> 2013-07-17 06:07:20,585 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-07-17 06:07:20,588 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-07-17 06:07:20,590 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Updating the current master key for generating delegation tokens
> 2013-07-17 06:07:20,632 INFO org.apache.hadoop.mapred.JobTracker:
> Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2013-07-17 06:07:20,633 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-07-17 06:07:20,665 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Starting expired delegation token remover thread,
> tokenRemoverScanInterval=60 min(s)
> 2013-07-17 06:07:20,665 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Updating the current master key for generating delegation tokens
> 2013-07-17 06:07:20,997 INFO org.apache.hadoop.mapred.JobTracker: Starting
> jobtracker with owner as maedeh
> 2013-07-17 06:07:21,050 FATAL org.apache.hadoop.mapred.JobTracker:
> java.lang.RuntimeException: java.lang.RuntimeException: class
> org.apache.hadoop.mapred.AdaptiveScheduler not
> org.apache.hadoop.mapred.TaskScheduler
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2276)
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4956)
> Caused by: java.lang.RuntimeException: class
> org.apache.hadoop.mapred.AdaptiveScheduler not
> org.apache.hadoop.mapred.TaskScheduler
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:892)
> ... 5 more
>
> 2013-07-17 06:07:21,134 INFO org.apache.hadoop.mapred.JobTracker:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down JobTracker at master/192.168.0.112
> ************************************************************/
>
> how I can resolve this problem?please help me... my jar file is in this
> address: /home/maedeh/hadoop-0.20.203.0/lib/adaptivescheduler.jar