You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jagmohan Chauhan <si...@gmail.com> on 2013/03/03 08:58:23 UTC
Question regarding job execution
Hi
I am using Capacity scheduler on a cluster of 5 nodes. I submit 3 jobs to
the system with a single queue for same user. I am observing that they are
executed on FIFO basis even if the map slots are empty. According to my
observation , when the first job for the user ends then its 2nd job starts.
Is it true that when using Capacity Scheduler, for any user, its next job
will be executed when its prior job is finished?
--
Thanks and Regards
Jagmohan Chauhan
MSc student,CS
Univ. of Saskatchewan
IEEE Graduate Student Member
http://homepage.usask.ca/~jac735/
Re: Question regarding job execution
Posted by Harsh J <ha...@cloudera.com>.
I am assuming you refer to the YARN's CapacityScheduler.
The CS in YARN does support parallel job (the right term is application, or
'app', not 'job' anymore, when speaking in YARN's context) execution. If
looking at code of CapacityScheduler.java and LeafQueue.java, you can
notice it iterate over all applications for node-update time assignments
(see nodeUpdate(…) call (via NODE_UPDATE event) in former and
assignContainers(…) call in the latter).
On Sun, Mar 3, 2013 at 1:28 PM, Jagmohan Chauhan <simplefundumnnit@gmail.com
> wrote:
> Hi
>
> I am using Capacity scheduler on a cluster of 5 nodes. I submit 3 jobs to
> the system with a single queue for same user. I am observing that they are
> executed on FIFO basis even if the map slots are empty. According to my
> observation , when the first job for the user ends then its 2nd job starts.
>
> Is it true that when using Capacity Scheduler, for any user, its next job
> will be executed when its prior job is finished?
>
>
>
> --
> Thanks and Regards
> Jagmohan Chauhan
> MSc student,CS
> Univ. of Saskatchewan
> IEEE Graduate Student Member
>
> http://homepage.usask.ca/~jac735/
>
>
--
Harsh J
Re: Question regarding job execution
Posted by Harsh J <ha...@cloudera.com>.
I am assuming you refer to the YARN's CapacityScheduler.
The CS in YARN does support parallel job (the right term is application, or
'app', not 'job' anymore, when speaking in YARN's context) execution. If
looking at code of CapacityScheduler.java and LeafQueue.java, you can
notice it iterate over all applications for node-update time assignments
(see nodeUpdate(…) call (via NODE_UPDATE event) in former and
assignContainers(…) call in the latter).
On Sun, Mar 3, 2013 at 1:28 PM, Jagmohan Chauhan <simplefundumnnit@gmail.com
> wrote:
> Hi
>
> I am using Capacity scheduler on a cluster of 5 nodes. I submit 3 jobs to
> the system with a single queue for same user. I am observing that they are
> executed on FIFO basis even if the map slots are empty. According to my
> observation , when the first job for the user ends then its 2nd job starts.
>
> Is it true that when using Capacity Scheduler, for any user, its next job
> will be executed when its prior job is finished?
>
>
>
> --
> Thanks and Regards
> Jagmohan Chauhan
> MSc student,CS
> Univ. of Saskatchewan
> IEEE Graduate Student Member
>
> http://homepage.usask.ca/~jac735/
>
>
--
Harsh J
Re: Question regarding job execution
Posted by Harsh J <ha...@cloudera.com>.
I am assuming you refer to the YARN's CapacityScheduler.
The CS in YARN does support parallel job (the right term is application, or
'app', not 'job' anymore, when speaking in YARN's context) execution. If
looking at code of CapacityScheduler.java and LeafQueue.java, you can
notice it iterate over all applications for node-update time assignments
(see nodeUpdate(…) call (via NODE_UPDATE event) in former and
assignContainers(…) call in the latter).
On Sun, Mar 3, 2013 at 1:28 PM, Jagmohan Chauhan <simplefundumnnit@gmail.com
> wrote:
> Hi
>
> I am using Capacity scheduler on a cluster of 5 nodes. I submit 3 jobs to
> the system with a single queue for same user. I am observing that they are
> executed on FIFO basis even if the map slots are empty. According to my
> observation , when the first job for the user ends then its 2nd job starts.
>
> Is it true that when using Capacity Scheduler, for any user, its next job
> will be executed when its prior job is finished?
>
>
>
> --
> Thanks and Regards
> Jagmohan Chauhan
> MSc student,CS
> Univ. of Saskatchewan
> IEEE Graduate Student Member
>
> http://homepage.usask.ca/~jac735/
>
>
--
Harsh J
Re: Question regarding job execution
Posted by Harsh J <ha...@cloudera.com>.
I am assuming you refer to the YARN's CapacityScheduler.
The CS in YARN does support parallel job (the right term is application, or
'app', not 'job' anymore, when speaking in YARN's context) execution. If
looking at code of CapacityScheduler.java and LeafQueue.java, you can
notice it iterate over all applications for node-update time assignments
(see nodeUpdate(…) call (via NODE_UPDATE event) in former and
assignContainers(…) call in the latter).
On Sun, Mar 3, 2013 at 1:28 PM, Jagmohan Chauhan <simplefundumnnit@gmail.com
> wrote:
> Hi
>
> I am using Capacity scheduler on a cluster of 5 nodes. I submit 3 jobs to
> the system with a single queue for same user. I am observing that they are
> executed on FIFO basis even if the map slots are empty. According to my
> observation , when the first job for the user ends then its 2nd job starts.
>
> Is it true that when using Capacity Scheduler, for any user, its next job
> will be executed when its prior job is finished?
>
>
>
> --
> Thanks and Regards
> Jagmohan Chauhan
> MSc student,CS
> Univ. of Saskatchewan
> IEEE Graduate Student Member
>
> http://homepage.usask.ca/~jac735/
>
>
--
Harsh J