You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Jagmohan Chauhan <si...@gmail.com> on 2013/03/03 09:11:01 UTC
Technical question on Capacity Scheduler.
Hi
I am going through the Capacity Scheduler implementation. There is one
thing i did not understand clearly.
1. Does the o ff-switch task refers to a task in which data has to be
fetched over the network. It means its not node-local ?
2. Does off-switch task includes only the tasks for which map input has to
be fetched from a node on a different rack across the switch or it also
includes task where data has to be fetched from another node on same rack
on same switch?
--
Thanks and Regards
Jagmohan Chauhan
MSc student,CS
Univ. of Saskatchewan
IEEE Graduate Student Member
http://homepage.usask.ca/~jac735/
Re: Technical question on Capacity Scheduler.
Posted by Arun C Murthy <ac...@hortonworks.com>.
That is true *only* if you do not have enough slots to run tasks for other jobs of the same user in the entire cluster.
On Mar 5, 2013, at 9:02 PM, Jagmohan Chauhan wrote:
> I think i need to re-frame the question a bit. Yes CS supports multiple
> jobs in parallel but of different users. If i am submitting 3 different
> jobs for a same user at the same time, they are always executed in a FIFO
> manner.
> Am i correct?
>
> On Tue, Mar 5, 2013 at 8:55 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> The CS does support running jobs in parallel. Are you observing just
>> the UI or are also noticing a FIFO behavior in logs where assignments
>> can be seen with timestamps?
>>
>> On Wed, Mar 6, 2013 at 9:03 AM, Jagmohan Chauhan
>> <simplefunduare .mnnit@gmail.com <si...@gmail.com>> wrote:
>>> Hi All
>>>
>>> Can someone please reply to my queries?
>>>
>>> On Sun, Mar 3, 2013 at 5:47 PM, Jagmohan Chauhan <
>> simplefundumnnit@gmail.com
>>>> wrote:
>>>
>>>> Thanks Harsh.
>>>>
>>>> I have a few more questions.
>>>>
>>>> Q1: I found it in my experiments using CS that for any user , its next
>> job
>>>> does not start until its current one is finished. Is it true and are
>> there
>>>> any exceptions and if true then why is it so? I I did not find any such
>>>> condition in the implementation of CS.
>>>>
>>>> Q2: The concept of reserved slots is true only if speculative execution
>>>> is on. Am i correct ? If yes,then the code dealing with reserved slots
>> wont
>>>> be executed if speculative execution is off?
>>>>
>>>> PS: I am working on MRv1.
>>>>
>>>>
>>>> On Sun, Mar 3, 2013 at 2:41 AM, Harsh J <ha...@cloudera.com> wrote:
>>>>
>>>>> On Sun, Mar 3, 2013 at 1:41 PM, Jagmohan Chauhan <
>>>>> simplefundumnnit@gmail.com
>>>>>> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I am going through the Capacity Scheduler implementation. There is
>> one
>>>>>> thing i did not understand clearly.
>>>>>>
>>>>>
>>>>> Are you reading the YARN CapacityScheduler or the older, MRv1 one? I'd
>>>>> suggest reading the newer one for any implementation or research goals,
>>>>> for
>>>>> it to be more current and future-applicable.
>>>>>
>>>>>
>>>>>> 1. Does the o ff-switch task refers to a task in which data has to be
>>>>>> fetched over the network. It means its not node-local ?
>>>>>>
>>>>>
>>>>> Off-switch would imply off-rack, i.e. not node local, nor rack-local.
>>>>>
>>>>>
>>>>>> 2. Does off-switch task includes only the tasks for which map input
>>>>> has to
>>>>>> be fetched from a node on a different rack across the switch or it
>> also
>>>>>> includes task where data has to be fetched from another node on same
>>>>> rack
>>>>>> on same switch?
>>>>>>
>>>>>
>>>>> A task's input split is generally supposed to define all locations of
>>>>> available inputs. If the CS is unable to schedule to any of those
>>>>> locations, nor their racks, then it schedules an off-rack (see above)
>> task
>>>>> which has to pull the input from a different rack.
>>>>>
>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards
>>>>>> Jagmohan Chauhan
>>>>>> MSc student,CS
>>>>>> Univ. of Saskatchewan
>>>>>> IEEE Graduate Student Member
>>>>>>
>>>>>> http://homepage.usask.ca/~jac735/
>>>>>>
>>>>>
>>>>> Feel free to post any further impl. related questions! :)
>>>>>
>>>>> --
>>>>> Harsh J
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards
>>>> Jagmohan Chauhan
>>>> MSc student,CS
>>>> Univ. of Saskatchewan
>>>> IEEE Graduate Student Member
>>>>
>>>> http://homepage.usask.ca/~jac735/
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards
>>> Jagmohan Chauhan
>>> MSc student,CS
>>> Univ. of Saskatchewan
>>> IEEE Graduate Student Member
>>>
>>> http://homepage.usask.ca/~jac735/
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> Thanks and Regards
> Jagmohan Chauhan
> MSc student,CS
> Univ. of Saskatchewan
> IEEE Graduate Student Member
>
> http://homepage.usask.ca/~jac735/
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Technical question on Capacity Scheduler.
Posted by Jagmohan Chauhan <si...@gmail.com>.
I am observing the logs.
On Tue, Mar 5, 2013 at 9:02 PM, Jagmohan Chauhan <simplefundumnnit@gmail.com
> wrote:
> I think i need to re-frame the question a bit. Yes CS supports multiple
> jobs in parallel but of different users. If i am submitting 3 different
> jobs for a same user at the same time, they are always executed in a FIFO
> manner.
> Am i correct?
>
> On Tue, Mar 5, 2013 at 8:55 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> The CS does support running jobs in parallel. Are you observing just
>> the UI or are also noticing a FIFO behavior in logs where assignments
>> can be seen with timestamps?
>>
>> On Wed, Mar 6, 2013 at 9:03 AM, Jagmohan Chauhan
>> <simplefunduare .mnnit@gmail.com <si...@gmail.com>> wrote:
>> > Hi All
>> >
>> > Can someone please reply to my queries?
>> >
>> > On Sun, Mar 3, 2013 at 5:47 PM, Jagmohan Chauhan <
>> simplefundumnnit@gmail.com
>> >> wrote:
>> >
>> >> Thanks Harsh.
>> >>
>> >> I have a few more questions.
>> >>
>> >> Q1: I found it in my experiments using CS that for any user , its next
>> job
>> >> does not start until its current one is finished. Is it true and are
>> there
>> >> any exceptions and if true then why is it so? I I did not find any
>> such
>> >> condition in the implementation of CS.
>> >>
>> >> Q2: The concept of reserved slots is true only if speculative
>> execution
>> >> is on. Am i correct ? If yes,then the code dealing with reserved slots
>> wont
>> >> be executed if speculative execution is off?
>> >>
>> >> PS: I am working on MRv1.
>> >>
>> >>
>> >> On Sun, Mar 3, 2013 at 2:41 AM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >>> On Sun, Mar 3, 2013 at 1:41 PM, Jagmohan Chauhan <
>> >>> simplefundumnnit@gmail.com
>> >>> > wrote:
>> >>>
>> >>> > Hi
>> >>> >
>> >>> > I am going through the Capacity Scheduler implementation. There is
>> one
>> >>> > thing i did not understand clearly.
>> >>> >
>> >>>
>> >>> Are you reading the YARN CapacityScheduler or the older, MRv1 one? I'd
>> >>> suggest reading the newer one for any implementation or research
>> goals,
>> >>> for
>> >>> it to be more current and future-applicable.
>> >>>
>> >>>
>> >>> > 1. Does the o ff-switch task refers to a task in which data has to
>> be
>> >>> > fetched over the network. It means its not node-local ?
>> >>> >
>> >>>
>> >>> Off-switch would imply off-rack, i.e. not node local, nor rack-local.
>> >>>
>> >>>
>> >>> > 2. Does off-switch task includes only the tasks for which map input
>> >>> has to
>> >>> > be fetched from a node on a different rack across the switch or it
>> also
>> >>> > includes task where data has to be fetched from another node on same
>> >>> rack
>> >>> > on same switch?
>> >>> >
>> >>>
>> >>> A task's input split is generally supposed to define all locations of
>> >>> available inputs. If the CS is unable to schedule to any of those
>> >>> locations, nor their racks, then it schedules an off-rack (see above)
>> task
>> >>> which has to pull the input from a different rack.
>> >>>
>> >>>
>> >>> >
>> >>> > --
>> >>> > Thanks and Regards
>> >>> > Jagmohan Chauhan
>> >>> > MSc student,CS
>> >>> > Univ. of Saskatchewan
>> >>> > IEEE Graduate Student Member
>> >>> >
>> >>> > http://homepage.usask.ca/~jac735/
>> >>> >
>> >>>
>> >>> Feel free to post any further impl. related questions! :)
>> >>>
>> >>> --
>> >>> Harsh J
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Thanks and Regards
>> >> Jagmohan Chauhan
>> >> MSc student,CS
>> >> Univ. of Saskatchewan
>> >> IEEE Graduate Student Member
>> >>
>> >> http://homepage.usask.ca/~jac735/
>> >>
>> >
>> >
>> >
>> > --
>> > Thanks and Regards
>> > Jagmohan Chauhan
>> > MSc student,CS
>> > Univ. of Saskatchewan
>> > IEEE Graduate Student Member
>> >
>> > http://homepage.usask.ca/~jac735/
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> Thanks and Regards
> Jagmohan Chauhan
> MSc student,CS
> Univ. of Saskatchewan
> IEEE Graduate Student Member
>
> http://homepage.usask.ca/~jac735/
>
--
Thanks and Regards
Jagmohan Chauhan
MSc student,CS
Univ. of Saskatchewan
IEEE Graduate Student Member
http://homepage.usask.ca/~jac735/
Re: Technical question on Capacity Scheduler.
Posted by Jagmohan Chauhan <si...@gmail.com>.
I think i need to re-frame the question a bit. Yes CS supports multiple
jobs in parallel but of different users. If i am submitting 3 different
jobs for a same user at the same time, they are always executed in a FIFO
manner.
Am i correct?
On Tue, Mar 5, 2013 at 8:55 PM, Harsh J <ha...@cloudera.com> wrote:
> The CS does support running jobs in parallel. Are you observing just
> the UI or are also noticing a FIFO behavior in logs where assignments
> can be seen with timestamps?
>
> On Wed, Mar 6, 2013 at 9:03 AM, Jagmohan Chauhan
> <simplefunduare .mnnit@gmail.com <si...@gmail.com>> wrote:
> > Hi All
> >
> > Can someone please reply to my queries?
> >
> > On Sun, Mar 3, 2013 at 5:47 PM, Jagmohan Chauhan <
> simplefundumnnit@gmail.com
> >> wrote:
> >
> >> Thanks Harsh.
> >>
> >> I have a few more questions.
> >>
> >> Q1: I found it in my experiments using CS that for any user , its next
> job
> >> does not start until its current one is finished. Is it true and are
> there
> >> any exceptions and if true then why is it so? I I did not find any such
> >> condition in the implementation of CS.
> >>
> >> Q2: The concept of reserved slots is true only if speculative execution
> >> is on. Am i correct ? If yes,then the code dealing with reserved slots
> wont
> >> be executed if speculative execution is off?
> >>
> >> PS: I am working on MRv1.
> >>
> >>
> >> On Sun, Mar 3, 2013 at 2:41 AM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >>> On Sun, Mar 3, 2013 at 1:41 PM, Jagmohan Chauhan <
> >>> simplefundumnnit@gmail.com
> >>> > wrote:
> >>>
> >>> > Hi
> >>> >
> >>> > I am going through the Capacity Scheduler implementation. There is
> one
> >>> > thing i did not understand clearly.
> >>> >
> >>>
> >>> Are you reading the YARN CapacityScheduler or the older, MRv1 one? I'd
> >>> suggest reading the newer one for any implementation or research goals,
> >>> for
> >>> it to be more current and future-applicable.
> >>>
> >>>
> >>> > 1. Does the o ff-switch task refers to a task in which data has to be
> >>> > fetched over the network. It means its not node-local ?
> >>> >
> >>>
> >>> Off-switch would imply off-rack, i.e. not node local, nor rack-local.
> >>>
> >>>
> >>> > 2. Does off-switch task includes only the tasks for which map input
> >>> has to
> >>> > be fetched from a node on a different rack across the switch or it
> also
> >>> > includes task where data has to be fetched from another node on same
> >>> rack
> >>> > on same switch?
> >>> >
> >>>
> >>> A task's input split is generally supposed to define all locations of
> >>> available inputs. If the CS is unable to schedule to any of those
> >>> locations, nor their racks, then it schedules an off-rack (see above)
> task
> >>> which has to pull the input from a different rack.
> >>>
> >>>
> >>> >
> >>> > --
> >>> > Thanks and Regards
> >>> > Jagmohan Chauhan
> >>> > MSc student,CS
> >>> > Univ. of Saskatchewan
> >>> > IEEE Graduate Student Member
> >>> >
> >>> > http://homepage.usask.ca/~jac735/
> >>> >
> >>>
> >>> Feel free to post any further impl. related questions! :)
> >>>
> >>> --
> >>> Harsh J
> >>>
> >>
> >>
> >>
> >> --
> >> Thanks and Regards
> >> Jagmohan Chauhan
> >> MSc student,CS
> >> Univ. of Saskatchewan
> >> IEEE Graduate Student Member
> >>
> >> http://homepage.usask.ca/~jac735/
> >>
> >
> >
> >
> > --
> > Thanks and Regards
> > Jagmohan Chauhan
> > MSc student,CS
> > Univ. of Saskatchewan
> > IEEE Graduate Student Member
> >
> > http://homepage.usask.ca/~jac735/
>
>
>
> --
> Harsh J
>
--
Thanks and Regards
Jagmohan Chauhan
MSc student,CS
Univ. of Saskatchewan
IEEE Graduate Student Member
http://homepage.usask.ca/~jac735/
Re: Technical question on Capacity Scheduler.
Posted by Harsh J <ha...@cloudera.com>.
The CS does support running jobs in parallel. Are you observing just
the UI or are also noticing a FIFO behavior in logs where assignments
can be seen with timestamps?
On Wed, Mar 6, 2013 at 9:03 AM, Jagmohan Chauhan
<si...@gmail.com> wrote:
> Hi All
>
> Can someone please reply to my queries?
>
> On Sun, Mar 3, 2013 at 5:47 PM, Jagmohan Chauhan <simplefundumnnit@gmail.com
>> wrote:
>
>> Thanks Harsh.
>>
>> I have a few more questions.
>>
>> Q1: I found it in my experiments using CS that for any user , its next job
>> does not start until its current one is finished. Is it true and are there
>> any exceptions and if true then why is it so? I I did not find any such
>> condition in the implementation of CS.
>>
>> Q2: The concept of reserved slots is true only if speculative execution
>> is on. Am i correct ? If yes,then the code dealing with reserved slots wont
>> be executed if speculative execution is off?
>>
>> PS: I am working on MRv1.
>>
>>
>> On Sun, Mar 3, 2013 at 2:41 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> On Sun, Mar 3, 2013 at 1:41 PM, Jagmohan Chauhan <
>>> simplefundumnnit@gmail.com
>>> > wrote:
>>>
>>> > Hi
>>> >
>>> > I am going through the Capacity Scheduler implementation. There is one
>>> > thing i did not understand clearly.
>>> >
>>>
>>> Are you reading the YARN CapacityScheduler or the older, MRv1 one? I'd
>>> suggest reading the newer one for any implementation or research goals,
>>> for
>>> it to be more current and future-applicable.
>>>
>>>
>>> > 1. Does the o ff-switch task refers to a task in which data has to be
>>> > fetched over the network. It means its not node-local ?
>>> >
>>>
>>> Off-switch would imply off-rack, i.e. not node local, nor rack-local.
>>>
>>>
>>> > 2. Does off-switch task includes only the tasks for which map input
>>> has to
>>> > be fetched from a node on a different rack across the switch or it also
>>> > includes task where data has to be fetched from another node on same
>>> rack
>>> > on same switch?
>>> >
>>>
>>> A task's input split is generally supposed to define all locations of
>>> available inputs. If the CS is unable to schedule to any of those
>>> locations, nor their racks, then it schedules an off-rack (see above) task
>>> which has to pull the input from a different rack.
>>>
>>>
>>> >
>>> > --
>>> > Thanks and Regards
>>> > Jagmohan Chauhan
>>> > MSc student,CS
>>> > Univ. of Saskatchewan
>>> > IEEE Graduate Student Member
>>> >
>>> > http://homepage.usask.ca/~jac735/
>>> >
>>>
>>> Feel free to post any further impl. related questions! :)
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>>
>> --
>> Thanks and Regards
>> Jagmohan Chauhan
>> MSc student,CS
>> Univ. of Saskatchewan
>> IEEE Graduate Student Member
>>
>> http://homepage.usask.ca/~jac735/
>>
>
>
>
> --
> Thanks and Regards
> Jagmohan Chauhan
> MSc student,CS
> Univ. of Saskatchewan
> IEEE Graduate Student Member
>
> http://homepage.usask.ca/~jac735/
--
Harsh J
Re: Technical question on Capacity Scheduler.
Posted by Jagmohan Chauhan <si...@gmail.com>.
Hi All
Can someone please reply to my queries?
On Sun, Mar 3, 2013 at 5:47 PM, Jagmohan Chauhan <simplefundumnnit@gmail.com
> wrote:
> Thanks Harsh.
>
> I have a few more questions.
>
> Q1: I found it in my experiments using CS that for any user , its next job
> does not start until its current one is finished. Is it true and are there
> any exceptions and if true then why is it so? I I did not find any such
> condition in the implementation of CS.
>
> Q2: The concept of reserved slots is true only if speculative execution
> is on. Am i correct ? If yes,then the code dealing with reserved slots wont
> be executed if speculative execution is off?
>
> PS: I am working on MRv1.
>
>
> On Sun, Mar 3, 2013 at 2:41 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> On Sun, Mar 3, 2013 at 1:41 PM, Jagmohan Chauhan <
>> simplefundumnnit@gmail.com
>> > wrote:
>>
>> > Hi
>> >
>> > I am going through the Capacity Scheduler implementation. There is one
>> > thing i did not understand clearly.
>> >
>>
>> Are you reading the YARN CapacityScheduler or the older, MRv1 one? I'd
>> suggest reading the newer one for any implementation or research goals,
>> for
>> it to be more current and future-applicable.
>>
>>
>> > 1. Does the o ff-switch task refers to a task in which data has to be
>> > fetched over the network. It means its not node-local ?
>> >
>>
>> Off-switch would imply off-rack, i.e. not node local, nor rack-local.
>>
>>
>> > 2. Does off-switch task includes only the tasks for which map input
>> has to
>> > be fetched from a node on a different rack across the switch or it also
>> > includes task where data has to be fetched from another node on same
>> rack
>> > on same switch?
>> >
>>
>> A task's input split is generally supposed to define all locations of
>> available inputs. If the CS is unable to schedule to any of those
>> locations, nor their racks, then it schedules an off-rack (see above) task
>> which has to pull the input from a different rack.
>>
>>
>> >
>> > --
>> > Thanks and Regards
>> > Jagmohan Chauhan
>> > MSc student,CS
>> > Univ. of Saskatchewan
>> > IEEE Graduate Student Member
>> >
>> > http://homepage.usask.ca/~jac735/
>> >
>>
>> Feel free to post any further impl. related questions! :)
>>
>> --
>> Harsh J
>>
>
>
>
> --
> Thanks and Regards
> Jagmohan Chauhan
> MSc student,CS
> Univ. of Saskatchewan
> IEEE Graduate Student Member
>
> http://homepage.usask.ca/~jac735/
>
--
Thanks and Regards
Jagmohan Chauhan
MSc student,CS
Univ. of Saskatchewan
IEEE Graduate Student Member
http://homepage.usask.ca/~jac735/
Re: Technical question on Capacity Scheduler.
Posted by Jagmohan Chauhan <si...@gmail.com>.
Thanks Harsh.
I have a few more questions.
Q1: I found it in my experiments using CS that for any user , its next job
does not start until its current one is finished. Is it true and are there
any exceptions and if true then why is it so? I I did not find any such
condition in the implementation of CS.
Q2: The concept of reserved slots is true only if speculative execution is
on. Am i correct ? If yes,then the code dealing with reserved slots wont be
executed if speculative execution is off?
PS: I am working on MRv1.
On Sun, Mar 3, 2013 at 2:41 AM, Harsh J <ha...@cloudera.com> wrote:
> On Sun, Mar 3, 2013 at 1:41 PM, Jagmohan Chauhan <
> simplefundumnnit@gmail.com
> > wrote:
>
> > Hi
> >
> > I am going through the Capacity Scheduler implementation. There is one
> > thing i did not understand clearly.
> >
>
> Are you reading the YARN CapacityScheduler or the older, MRv1 one? I'd
> suggest reading the newer one for any implementation or research goals, for
> it to be more current and future-applicable.
>
>
> > 1. Does the o ff-switch task refers to a task in which data has to be
> > fetched over the network. It means its not node-local ?
> >
>
> Off-switch would imply off-rack, i.e. not node local, nor rack-local.
>
>
> > 2. Does off-switch task includes only the tasks for which map input has
> to
> > be fetched from a node on a different rack across the switch or it also
> > includes task where data has to be fetched from another node on same rack
> > on same switch?
> >
>
> A task's input split is generally supposed to define all locations of
> available inputs. If the CS is unable to schedule to any of those
> locations, nor their racks, then it schedules an off-rack (see above) task
> which has to pull the input from a different rack.
>
>
> >
> > --
> > Thanks and Regards
> > Jagmohan Chauhan
> > MSc student,CS
> > Univ. of Saskatchewan
> > IEEE Graduate Student Member
> >
> > http://homepage.usask.ca/~jac735/
> >
>
> Feel free to post any further impl. related questions! :)
>
> --
> Harsh J
>
--
Thanks and Regards
Jagmohan Chauhan
MSc student,CS
Univ. of Saskatchewan
IEEE Graduate Student Member
http://homepage.usask.ca/~jac735/
Re: Technical question on Capacity Scheduler.
Posted by Harsh J <ha...@cloudera.com>.
On Sun, Mar 3, 2013 at 1:41 PM, Jagmohan Chauhan <simplefundumnnit@gmail.com
> wrote:
> Hi
>
> I am going through the Capacity Scheduler implementation. There is one
> thing i did not understand clearly.
>
Are you reading the YARN CapacityScheduler or the older, MRv1 one? I'd
suggest reading the newer one for any implementation or research goals, for
it to be more current and future-applicable.
> 1. Does the o ff-switch task refers to a task in which data has to be
> fetched over the network. It means its not node-local ?
>
Off-switch would imply off-rack, i.e. not node local, nor rack-local.
> 2. Does off-switch task includes only the tasks for which map input has to
> be fetched from a node on a different rack across the switch or it also
> includes task where data has to be fetched from another node on same rack
> on same switch?
>
A task's input split is generally supposed to define all locations of
available inputs. If the CS is unable to schedule to any of those
locations, nor their racks, then it schedules an off-rack (see above) task
which has to pull the input from a different rack.
>
> --
> Thanks and Regards
> Jagmohan Chauhan
> MSc student,CS
> Univ. of Saskatchewan
> IEEE Graduate Student Member
>
> http://homepage.usask.ca/~jac735/
>
Feel free to post any further impl. related questions! :)
--
Harsh J