You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Sreejith Ramakrishnan <sr...@gmail.com> on 2013/07/11 08:49:47 UTC
My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
I'm making my own scheduler. Since I had errors, I rewrote the scheduler to
do just a simple thing. When I give it a job, it allocates 1 map() and 1
reduce() to it. But that isn't happening. I used a barebones WordCount pgm.
I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split for
<my-job-id>
DETAILED INFO:
Here's some snippets of code. The scheduler logic is inside the
assignTasks():
We get a JobInProgressListener in the constructor-
this.jobQueueJobInProgressListener = new JobQueueJobInProgressListener();
In start() -
this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgressListener);
Then I just take the last element in that queue (I assume its the latest
submitted job) and assign to incomingJob (of type JobInProgress).
Then I obtain a new map task as follows and append this task to a list
which is the return value of the assignTasks():
Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus,
numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
I've attached the scheduler. Can any expert point out if there's any
ignorant mistakes from my side?
RE: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
Posted by Devaraj k <de...@huawei.com>.
It seems tasks initialization is not happening for the Job. Do you see any other message or error in the JT log related to that Job/Job-Id.
Thanks
Devaraj k
From: Sreejith Ramakrishnan [mailto:sreejith.code@gmail.com]
Sent: 11 July 2013 12:20
To: mapreduce-dev@hadoop.apache.org
Subject: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
I'm making my own scheduler. Since I had errors, I rewrote the scheduler to do just a simple thing. When I give it a job, it allocates 1 map() and 1 reduce() to it. But that isn't happening. I used a barebones WordCount pgm. I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split for <my-job-id>
DETAILED INFO:
Here's some snippets of code. The scheduler logic is inside the assignTasks():
We get a JobInProgressListener in the constructor-
this.jobQueueJobInProgressListener = new JobQueueJobInProgressListener();
In start() -
this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgressListener);
Then I just take the last element in that queue (I assume its the latest submitted job) and assign to incomingJob (of type JobInProgress).
Then I obtain a new map task as follows and append this task to a list which is the return value of the assignTasks():
Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus, numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
I've attached the scheduler. Can any expert point out if there's any ignorant mistakes from my side?
Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
Posted by Sreejith Ramakrishnan <sr...@gmail.com>.
Ya. I'll be doing that. Meantime, do you know how control flows and
reaches assignTasks()?
If so, please do tell me. I'm suspecting a gap in my understanding.
On 11-Jul-2013 6:48 PM, "Devaraj k" <de...@gmail.com> wrote:
> Could you debug and check the Job Tracker/your scheduler code, then you can
> pinpoint the exact issue and fix it.
>
> Thanks
> Devaraj k
>
>
> On Thu, Jul 11, 2013 at 7:58 PM, Sreejith Ramakrishnan <
> sreejith.code@gmail.com> wrote:
>
> > @Devaraj K
> >
> > I tried setting mapred.jobtracker.maxtasks.per.job to 4 in
> mapred-site.xml.
> > In the subsequent job, I got this in the log which clearly shows the
> > configuration has taken effect:
> >
> > INFO org.apache.hadoop.mapred.JobInProgress: job_201307111721_0001:
> nMaps=1
> > nReduces=1 max=4
> >
> > Yet, the original problem remains. It doesn't go anywhere from 0%. Btw,
> > when we get the jobQueue, should we specify a name for the queue we want?
> > (eg. default)
> >
> > INFO org.apache.hadoop.mapred.JobTracker: Job job_201307111721_0001 added
> > successfully for user 'sreejith' to queue 'default'
> >
> > Also, is there a limit to the no.of tasks which can be obtained in a
> > heartbeat?
> >
> >
> > On Thu, Jul 11, 2013 at 2:15 PM, Devaraj k <de...@huawei.com> wrote:
> >
> > > You don't need to explicitly call the initTasks(), you can find this
> > > invocation in JobTracker.java class.
> > >
> > > >*Yet the **Cannot create task split for <my-job-id>** is still there*.
> > > >About errors relating to the JobID, is this significant? --- INFO
> > > >org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
> > > >nReduces=1 max=-1
> > >
> > > Here it shows max=-1, which means max configured(i.e default) tasks for
> > > the Job is -1.
> > >
> > > I think you can see this exception from JT log.
> > >
> > > LOG.info(jobId + ": nMaps=" + numMapTasks + " nReduces=" +
> numReduceTasks
> > > + " max=" + maxTasks);
> > > if (maxTasks > 0 && (numMapTasks + numReduceTasks) > maxTasks) {
> > > throw new IOException(
> > > "The number of tasks for this job " +
> > > (numMapTasks + numReduceTasks) +
> > > " exceeds the configured limit " + maxTasks);
> > > }
> > >
> > > Can you try configuring some number for this configuration
> > > 'mapred.jobtracker.maxtasks.per.job' and run the same Job?
> > >
> > >
> > > Thanks
> > > Devaraj k
> > >
> > >
> > > -----Original Message-----
> > > From: Sreejith Ramakrishnan [mailto:sreejith.code@gmail.com]
> > > Sent: 11 July 2013 13:26
> > > To: mapreduce-dev@hadoop.apache.org
> > > Subject: Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce
> > 0%
> > >
> > > @Devaraj K
> > >
> > > *I tried explicitly calling incomingJob.initTasks()*and it gave me a
> > > pleasant INFO org.apache.hadoop.mapred.JobInProgress: Job
> > > job_201307111248_0002 initialized successfully with 1 map tasks and 1
> > > reduce tasks
> > >
> > > *Yet the **Cannot create task split for <my-job-id>** is still there*.
> > > About errors relating to the JobID, is this significant? --- INFO
> > > org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
> > > nReduces=1 max=-1
> > >
> > > *P.S: In the log file I shared, anything with "sreejith == " were
> logged
> > > by me. Not system-generated*
> > >
> > >
> > > On Thu, Jul 11, 2013 at 1:18 PM, Sreejith Ramakrishnan <
> > > sreejith.code@gmail.com> wrote:
> > >
> > > > I've attached the relevant portion of the log.
> > > >
> > > > For easy testing, *I've attached the maven project* for the
> > > > EDFJobScheduler. Just make a JAR from it and put it in
> > > > $HADOOP_HOME/lib and add this property to *mapred-site.xml*:
> > > >
> > > > *<property>
> > > > *
> > > > * <name>mapred.jobtracker.taskScheduler</name>
> > > > <value>org.apache.hadoop.mapred.EDFJobScheduler</value>
> > > > *
> > > > *</property>*
> > > >
> > > >
> > > > On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan <
> > > > sreejith.code@gmail.com> wrote:
> > > >
> > > >> I'm making my own scheduler. Since I had errors, I rewrote the
> > > >> scheduler to do just a simple thing. When I give it a job, it
> > > >> allocates 1 map() and 1
> > > >> reduce() to it. But that isn't happening. I used a barebones
> WordCount
> > > pgm.
> > > >> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
> > > >>
> > > >> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task
> split
> > > >> for <my-job-id>
> > > >>
> > > >> DETAILED INFO:
> > > >>
> > > >> Here's some snippets of code. The scheduler logic is inside the
> > > >> assignTasks():
> > > >>
> > > >> We get a JobInProgressListener in the constructor-
> > > >> this.jobQueueJobInProgressListener = new
> > > >> JobQueueJobInProgressListener();
> > > >>
> > > >> In start() -
> > > >>
> > > >>
> this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgres
> > > >> sListener);
> > > >>
> > > >> Then I just take the last element in that queue (I assume its the
> > > >> latest submitted job) and assign to incomingJob (of type
> > JobInProgress).
> > > >>
> > > >> Then I obtain a new map task as follows and append this task to a
> > > >> list which is the return value of the assignTasks():
> > > >> Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus,
> > > >> numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
> > > >>
> > > >>
> > > >> I've attached the scheduler. Can any expert point out if there's any
> > > >> ignorant mistakes from my side?
> > > >>
> > > >
> > > >
> > >
> >
>
Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
Posted by Devaraj k <de...@gmail.com>.
Could you debug and check the Job Tracker/your scheduler code, then you can
pinpoint the exact issue and fix it.
Thanks
Devaraj k
On Thu, Jul 11, 2013 at 7:58 PM, Sreejith Ramakrishnan <
sreejith.code@gmail.com> wrote:
> @Devaraj K
>
> I tried setting mapred.jobtracker.maxtasks.per.job to 4 in mapred-site.xml.
> In the subsequent job, I got this in the log which clearly shows the
> configuration has taken effect:
>
> INFO org.apache.hadoop.mapred.JobInProgress: job_201307111721_0001: nMaps=1
> nReduces=1 max=4
>
> Yet, the original problem remains. It doesn't go anywhere from 0%. Btw,
> when we get the jobQueue, should we specify a name for the queue we want?
> (eg. default)
>
> INFO org.apache.hadoop.mapred.JobTracker: Job job_201307111721_0001 added
> successfully for user 'sreejith' to queue 'default'
>
> Also, is there a limit to the no.of tasks which can be obtained in a
> heartbeat?
>
>
> On Thu, Jul 11, 2013 at 2:15 PM, Devaraj k <de...@huawei.com> wrote:
>
> > You don't need to explicitly call the initTasks(), you can find this
> > invocation in JobTracker.java class.
> >
> > >*Yet the **Cannot create task split for <my-job-id>** is still there*.
> > >About errors relating to the JobID, is this significant? --- INFO
> > >org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
> > >nReduces=1 max=-1
> >
> > Here it shows max=-1, which means max configured(i.e default) tasks for
> > the Job is -1.
> >
> > I think you can see this exception from JT log.
> >
> > LOG.info(jobId + ": nMaps=" + numMapTasks + " nReduces=" + numReduceTasks
> > + " max=" + maxTasks);
> > if (maxTasks > 0 && (numMapTasks + numReduceTasks) > maxTasks) {
> > throw new IOException(
> > "The number of tasks for this job " +
> > (numMapTasks + numReduceTasks) +
> > " exceeds the configured limit " + maxTasks);
> > }
> >
> > Can you try configuring some number for this configuration
> > 'mapred.jobtracker.maxtasks.per.job' and run the same Job?
> >
> >
> > Thanks
> > Devaraj k
> >
> >
> > -----Original Message-----
> > From: Sreejith Ramakrishnan [mailto:sreejith.code@gmail.com]
> > Sent: 11 July 2013 13:26
> > To: mapreduce-dev@hadoop.apache.org
> > Subject: Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce
> 0%
> >
> > @Devaraj K
> >
> > *I tried explicitly calling incomingJob.initTasks()*and it gave me a
> > pleasant INFO org.apache.hadoop.mapred.JobInProgress: Job
> > job_201307111248_0002 initialized successfully with 1 map tasks and 1
> > reduce tasks
> >
> > *Yet the **Cannot create task split for <my-job-id>** is still there*.
> > About errors relating to the JobID, is this significant? --- INFO
> > org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
> > nReduces=1 max=-1
> >
> > *P.S: In the log file I shared, anything with "sreejith == " were logged
> > by me. Not system-generated*
> >
> >
> > On Thu, Jul 11, 2013 at 1:18 PM, Sreejith Ramakrishnan <
> > sreejith.code@gmail.com> wrote:
> >
> > > I've attached the relevant portion of the log.
> > >
> > > For easy testing, *I've attached the maven project* for the
> > > EDFJobScheduler. Just make a JAR from it and put it in
> > > $HADOOP_HOME/lib and add this property to *mapred-site.xml*:
> > >
> > > *<property>
> > > *
> > > * <name>mapred.jobtracker.taskScheduler</name>
> > > <value>org.apache.hadoop.mapred.EDFJobScheduler</value>
> > > *
> > > *</property>*
> > >
> > >
> > > On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan <
> > > sreejith.code@gmail.com> wrote:
> > >
> > >> I'm making my own scheduler. Since I had errors, I rewrote the
> > >> scheduler to do just a simple thing. When I give it a job, it
> > >> allocates 1 map() and 1
> > >> reduce() to it. But that isn't happening. I used a barebones WordCount
> > pgm.
> > >> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
> > >>
> > >> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split
> > >> for <my-job-id>
> > >>
> > >> DETAILED INFO:
> > >>
> > >> Here's some snippets of code. The scheduler logic is inside the
> > >> assignTasks():
> > >>
> > >> We get a JobInProgressListener in the constructor-
> > >> this.jobQueueJobInProgressListener = new
> > >> JobQueueJobInProgressListener();
> > >>
> > >> In start() -
> > >>
> > >> this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgres
> > >> sListener);
> > >>
> > >> Then I just take the last element in that queue (I assume its the
> > >> latest submitted job) and assign to incomingJob (of type
> JobInProgress).
> > >>
> > >> Then I obtain a new map task as follows and append this task to a
> > >> list which is the return value of the assignTasks():
> > >> Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus,
> > >> numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
> > >>
> > >>
> > >> I've attached the scheduler. Can any expert point out if there's any
> > >> ignorant mistakes from my side?
> > >>
> > >
> > >
> >
>
Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
Posted by Sreejith Ramakrishnan <sr...@gmail.com>.
@Devaraj K
I tried setting mapred.jobtracker.maxtasks.per.job to 4 in mapred-site.xml.
In the subsequent job, I got this in the log which clearly shows the
configuration has taken effect:
INFO org.apache.hadoop.mapred.JobInProgress: job_201307111721_0001: nMaps=1
nReduces=1 max=4
Yet, the original problem remains. It doesn't go anywhere from 0%. Btw,
when we get the jobQueue, should we specify a name for the queue we want?
(eg. default)
INFO org.apache.hadoop.mapred.JobTracker: Job job_201307111721_0001 added
successfully for user 'sreejith' to queue 'default'
Also, is there a limit to the no.of tasks which can be obtained in a
heartbeat?
On Thu, Jul 11, 2013 at 2:15 PM, Devaraj k <de...@huawei.com> wrote:
> You don't need to explicitly call the initTasks(), you can find this
> invocation in JobTracker.java class.
>
> >*Yet the **Cannot create task split for <my-job-id>** is still there*.
> >About errors relating to the JobID, is this significant? --- INFO
> >org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
> >nReduces=1 max=-1
>
> Here it shows max=-1, which means max configured(i.e default) tasks for
> the Job is -1.
>
> I think you can see this exception from JT log.
>
> LOG.info(jobId + ": nMaps=" + numMapTasks + " nReduces=" + numReduceTasks
> + " max=" + maxTasks);
> if (maxTasks > 0 && (numMapTasks + numReduceTasks) > maxTasks) {
> throw new IOException(
> "The number of tasks for this job " +
> (numMapTasks + numReduceTasks) +
> " exceeds the configured limit " + maxTasks);
> }
>
> Can you try configuring some number for this configuration
> 'mapred.jobtracker.maxtasks.per.job' and run the same Job?
>
>
> Thanks
> Devaraj k
>
>
> -----Original Message-----
> From: Sreejith Ramakrishnan [mailto:sreejith.code@gmail.com]
> Sent: 11 July 2013 13:26
> To: mapreduce-dev@hadoop.apache.org
> Subject: Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
>
> @Devaraj K
>
> *I tried explicitly calling incomingJob.initTasks()*and it gave me a
> pleasant INFO org.apache.hadoop.mapred.JobInProgress: Job
> job_201307111248_0002 initialized successfully with 1 map tasks and 1
> reduce tasks
>
> *Yet the **Cannot create task split for <my-job-id>** is still there*.
> About errors relating to the JobID, is this significant? --- INFO
> org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
> nReduces=1 max=-1
>
> *P.S: In the log file I shared, anything with "sreejith == " were logged
> by me. Not system-generated*
>
>
> On Thu, Jul 11, 2013 at 1:18 PM, Sreejith Ramakrishnan <
> sreejith.code@gmail.com> wrote:
>
> > I've attached the relevant portion of the log.
> >
> > For easy testing, *I've attached the maven project* for the
> > EDFJobScheduler. Just make a JAR from it and put it in
> > $HADOOP_HOME/lib and add this property to *mapred-site.xml*:
> >
> > *<property>
> > *
> > * <name>mapred.jobtracker.taskScheduler</name>
> > <value>org.apache.hadoop.mapred.EDFJobScheduler</value>
> > *
> > *</property>*
> >
> >
> > On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan <
> > sreejith.code@gmail.com> wrote:
> >
> >> I'm making my own scheduler. Since I had errors, I rewrote the
> >> scheduler to do just a simple thing. When I give it a job, it
> >> allocates 1 map() and 1
> >> reduce() to it. But that isn't happening. I used a barebones WordCount
> pgm.
> >> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
> >>
> >> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split
> >> for <my-job-id>
> >>
> >> DETAILED INFO:
> >>
> >> Here's some snippets of code. The scheduler logic is inside the
> >> assignTasks():
> >>
> >> We get a JobInProgressListener in the constructor-
> >> this.jobQueueJobInProgressListener = new
> >> JobQueueJobInProgressListener();
> >>
> >> In start() -
> >>
> >> this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgres
> >> sListener);
> >>
> >> Then I just take the last element in that queue (I assume its the
> >> latest submitted job) and assign to incomingJob (of type JobInProgress).
> >>
> >> Then I obtain a new map task as follows and append this task to a
> >> list which is the return value of the assignTasks():
> >> Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus,
> >> numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
> >>
> >>
> >> I've attached the scheduler. Can any expert point out if there's any
> >> ignorant mistakes from my side?
> >>
> >
> >
>
RE: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
Posted by Devaraj k <de...@huawei.com>.
You don't need to explicitly call the initTasks(), you can find this invocation in JobTracker.java class.
>*Yet the **Cannot create task split for <my-job-id>** is still there*.
>About errors relating to the JobID, is this significant? --- INFO
>org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
>nReduces=1 max=-1
Here it shows max=-1, which means max configured(i.e default) tasks for the Job is -1.
I think you can see this exception from JT log.
LOG.info(jobId + ": nMaps=" + numMapTasks + " nReduces=" + numReduceTasks + " max=" + maxTasks);
if (maxTasks > 0 && (numMapTasks + numReduceTasks) > maxTasks) {
throw new IOException(
"The number of tasks for this job " +
(numMapTasks + numReduceTasks) +
" exceeds the configured limit " + maxTasks);
}
Can you try configuring some number for this configuration 'mapred.jobtracker.maxtasks.per.job' and run the same Job?
Thanks
Devaraj k
-----Original Message-----
From: Sreejith Ramakrishnan [mailto:sreejith.code@gmail.com]
Sent: 11 July 2013 13:26
To: mapreduce-dev@hadoop.apache.org
Subject: Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
@Devaraj K
*I tried explicitly calling incomingJob.initTasks()*and it gave me a pleasant INFO org.apache.hadoop.mapred.JobInProgress: Job
job_201307111248_0002 initialized successfully with 1 map tasks and 1 reduce tasks
*Yet the **Cannot create task split for <my-job-id>** is still there*.
About errors relating to the JobID, is this significant? --- INFO
org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
nReduces=1 max=-1
*P.S: In the log file I shared, anything with "sreejith == " were logged by me. Not system-generated*
On Thu, Jul 11, 2013 at 1:18 PM, Sreejith Ramakrishnan < sreejith.code@gmail.com> wrote:
> I've attached the relevant portion of the log.
>
> For easy testing, *I've attached the maven project* for the
> EDFJobScheduler. Just make a JAR from it and put it in
> $HADOOP_HOME/lib and add this property to *mapred-site.xml*:
>
> *<property>
> *
> * <name>mapred.jobtracker.taskScheduler</name>
> <value>org.apache.hadoop.mapred.EDFJobScheduler</value>
> *
> *</property>*
>
>
> On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan <
> sreejith.code@gmail.com> wrote:
>
>> I'm making my own scheduler. Since I had errors, I rewrote the
>> scheduler to do just a simple thing. When I give it a job, it
>> allocates 1 map() and 1
>> reduce() to it. But that isn't happening. I used a barebones WordCount pgm.
>> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
>>
>> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split
>> for <my-job-id>
>>
>> DETAILED INFO:
>>
>> Here's some snippets of code. The scheduler logic is inside the
>> assignTasks():
>>
>> We get a JobInProgressListener in the constructor-
>> this.jobQueueJobInProgressListener = new
>> JobQueueJobInProgressListener();
>>
>> In start() -
>>
>> this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgres
>> sListener);
>>
>> Then I just take the last element in that queue (I assume its the
>> latest submitted job) and assign to incomingJob (of type JobInProgress).
>>
>> Then I obtain a new map task as follows and append this task to a
>> list which is the return value of the assignTasks():
>> Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus,
>> numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
>>
>>
>> I've attached the scheduler. Can any expert point out if there's any
>> ignorant mistakes from my side?
>>
>
>
Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
Posted by Sreejith Ramakrishnan <sr...@gmail.com>.
@Devaraj K
*I tried explicitly calling incomingJob.initTasks()*and it gave me a
pleasant INFO org.apache.hadoop.mapred.JobInProgress: Job
job_201307111248_0002 initialized successfully with 1 map tasks and 1
reduce tasks
*Yet the **Cannot create task split for <my-job-id>** is still there*.
About errors relating to the JobID, is this significant? --- INFO
org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
nReduces=1 max=-1
*P.S: In the log file I shared, anything with "sreejith == " were logged by
me. Not system-generated*
On Thu, Jul 11, 2013 at 1:18 PM, Sreejith Ramakrishnan <
sreejith.code@gmail.com> wrote:
> I've attached the relevant portion of the log.
>
> For easy testing, *I've attached the maven project* for the
> EDFJobScheduler. Just make a JAR from it and put it in $HADOOP_HOME/lib and
> add this property to *mapred-site.xml*:
>
> *<property>
> *
> * <name>mapred.jobtracker.taskScheduler</name>
> <value>org.apache.hadoop.mapred.EDFJobScheduler</value>
> *
> *</property>*
>
>
> On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan <
> sreejith.code@gmail.com> wrote:
>
>> I'm making my own scheduler. Since I had errors, I rewrote the scheduler
>> to do just a simple thing. When I give it a job, it allocates 1 map() and 1
>> reduce() to it. But that isn't happening. I used a barebones WordCount pgm.
>> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
>>
>> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split for
>> <my-job-id>
>>
>> DETAILED INFO:
>>
>> Here's some snippets of code. The scheduler logic is inside the
>> assignTasks():
>>
>> We get a JobInProgressListener in the constructor-
>> this.jobQueueJobInProgressListener = new JobQueueJobInProgressListener();
>>
>> In start() -
>>
>> this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgressListener);
>>
>> Then I just take the last element in that queue (I assume its the latest
>> submitted job) and assign to incomingJob (of type JobInProgress).
>>
>> Then I obtain a new map task as follows and append this task to a list
>> which is the return value of the assignTasks():
>> Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus,
>> numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
>>
>>
>> I've attached the scheduler. Can any expert point out if there's any
>> ignorant mistakes from my side?
>>
>
>
Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
Posted by Sreejith Ramakrishnan <sr...@gmail.com>.
I've attached the relevant portion of the log.
For easy testing, *I've attached the maven project* for the
EDFJobScheduler. Just make a JAR from it and put it in $HADOOP_HOME/lib and
add this property to *mapred-site.xml*:
*<property>
*
* <name>mapred.jobtracker.taskScheduler</name>
<value>org.apache.hadoop.mapred.EDFJobScheduler</value>
*
*</property>*
On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan <
sreejith.code@gmail.com> wrote:
> I'm making my own scheduler. Since I had errors, I rewrote the scheduler
> to do just a simple thing. When I give it a job, it allocates 1 map() and 1
> reduce() to it. But that isn't happening. I used a barebones WordCount pgm.
> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
>
> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split for
> <my-job-id>
>
> DETAILED INFO:
>
> Here's some snippets of code. The scheduler logic is inside the
> assignTasks():
>
> We get a JobInProgressListener in the constructor-
> this.jobQueueJobInProgressListener = new JobQueueJobInProgressListener();
>
> In start() -
>
> this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgressListener);
>
> Then I just take the last element in that queue (I assume its the latest
> submitted job) and assign to incomingJob (of type JobInProgress).
>
> Then I obtain a new map task as follows and append this task to a list
> which is the return value of the assignTasks():
> Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus,
> numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
>
>
> I've attached the scheduler. Can any expert point out if there's any
> ignorant mistakes from my side?
>