You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Shen LI <ge...@gmail.com> on 2010/10/07 20:57:51 UTC
How to modify task assignment algorithm?
Hi,
How can I modify the task assignment strategy in hadoop which is used to
assign tasks to different worker nodes? (Not the job scheduler)
Big thanks,
Shen
Re: How to modify task assignment algorithm?
Posted by Todd Lipcon <to...@cloudera.com>.
On Sun, Oct 10, 2010 at 7:22 PM, Ken Goodhope <ke...@gmail.com> wrote:
> Another option might be to extend fair scheduler's task selector. The
> code is already set up to make this a configurable abstraction point.
> One word of caution, if you are using the cloudera distro, you will
> find that fair scheduler has been modified to not use those classes
> anymore. Something that wasn't documented very well. Still works with
> 20.2. Not sure about 21.
Just to clarify, we don't have any custom non-Apache code in our fair
scheduler in CDH. Rather, we've backported newer fair scheduler work
from 0.21 and trunk into 0.20. Most importantly, we backported
MAPREDUCE-706 which changed a lot of the internals.
Thanks
-Todd
>
> On Thursday, October 7, 2010, Jeff Zhang <zj...@gmail.com> wrote:
>> Currently, TaskTracker send heatbeat to JobTracker, and JobTracker
>> will send actions to TaskTracker according TaskTracker's status. I
>> think you should do some hacking about the heatbeat part of JobTracker
>>
>> public synchronized HeartbeatResponse heartbeat(TaskTrackerStatus status,
>> boolean restarted,
>> boolean initialContact,
>> boolean acceptNewTasks,
>> short responseId)
>>
>>
>>
>> On Fri, Oct 8, 2010 at 10:38 AM, Shen LI <ge...@gmail.com> wrote:
>>> Hi, Thanks you very much for your reply. I want to run my own algorithm for
>>> this part to see if we can achieve better outcome in specific scenario. So
>>> how can I modify it?
>>> Thanks a lot!
>>> Shen
>>>
>>> On Thu, Oct 7, 2010 at 6:33 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>>>
>>>> I believe it is possible. But what is your purpose? I believe current
>>>> solution is good enough.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Oct 8, 2010 at 2:57 AM, Shen LI <ge...@gmail.com> wrote:
>>>> > Hi,
>>>> > How can I modify the task assignment strategy in hadoop which is used to
>>>> > assign tasks to different worker nodes? (Not the job scheduler)
>>>> > Big thanks,
>>>> > Shen
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards
>>>>
>>>> Jeff Zhang
>>>
>>>
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: How to modify task assignment algorithm?
Posted by Ken Goodhope <ke...@gmail.com>.
Another option might be to extend fair scheduler's task selector. The
code is already set up to make this a configurable abstraction point.
One word of caution, if you are using the cloudera distro, you will
find that fair scheduler has been modified to not use those classes
anymore. Something that wasn't documented very well. Still works with
20.2. Not sure about 21.
On Thursday, October 7, 2010, Jeff Zhang <zj...@gmail.com> wrote:
> Currently, TaskTracker send heatbeat to JobTracker, and JobTracker
> will send actions to TaskTracker according TaskTracker's status. I
> think you should do some hacking about the heatbeat part of JobTracker
>
> public synchronized HeartbeatResponse heartbeat(TaskTrackerStatus status,
> boolean restarted,
> boolean initialContact,
> boolean acceptNewTasks,
> short responseId)
>
>
>
> On Fri, Oct 8, 2010 at 10:38 AM, Shen LI <ge...@gmail.com> wrote:
>> Hi, Thanks you very much for your reply. I want to run my own algorithm for
>> this part to see if we can achieve better outcome in specific scenario. So
>> how can I modify it?
>> Thanks a lot!
>> Shen
>>
>> On Thu, Oct 7, 2010 at 6:33 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>>
>>> I believe it is possible. But what is your purpose? I believe current
>>> solution is good enough.
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Oct 8, 2010 at 2:57 AM, Shen LI <ge...@gmail.com> wrote:
>>> > Hi,
>>> > How can I modify the task assignment strategy in hadoop which is used to
>>> > assign tasks to different worker nodes? (Not the job scheduler)
>>> > Big thanks,
>>> > Shen
>>>
>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: How to modify task assignment algorithm?
Posted by Jeff Zhang <zj...@gmail.com>.
Currently, TaskTracker send heatbeat to JobTracker, and JobTracker
will send actions to TaskTracker according TaskTracker's status. I
think you should do some hacking about the heatbeat part of JobTracker
public synchronized HeartbeatResponse heartbeat(TaskTrackerStatus status,
boolean restarted,
boolean initialContact,
boolean acceptNewTasks,
short responseId)
On Fri, Oct 8, 2010 at 10:38 AM, Shen LI <ge...@gmail.com> wrote:
> Hi, Thanks you very much for your reply. I want to run my own algorithm for
> this part to see if we can achieve better outcome in specific scenario. So
> how can I modify it?
> Thanks a lot!
> Shen
>
> On Thu, Oct 7, 2010 at 6:33 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>
>> I believe it is possible. But what is your purpose? I believe current
>> solution is good enough.
>>
>>
>>
>>
>>
>> On Fri, Oct 8, 2010 at 2:57 AM, Shen LI <ge...@gmail.com> wrote:
>> > Hi,
>> > How can I modify the task assignment strategy in hadoop which is used to
>> > assign tasks to different worker nodes? (Not the job scheduler)
>> > Big thanks,
>> > Shen
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>
>
--
Best Regards
Jeff Zhang
Re: How to modify task assignment algorithm?
Posted by Shen LI <ge...@gmail.com>.
Hi, Thanks you very much for your reply. I want to run my own algorithm for
this part to see if we can achieve better outcome in specific scenario. So
how can I modify it?
Thanks a lot!
Shen
On Thu, Oct 7, 2010 at 6:33 PM, Jeff Zhang <zj...@gmail.com> wrote:
> I believe it is possible. But what is your purpose? I believe current
> solution is good enough.
>
>
>
>
>
> On Fri, Oct 8, 2010 at 2:57 AM, Shen LI <ge...@gmail.com> wrote:
> > Hi,
> > How can I modify the task assignment strategy in hadoop which is used to
> > assign tasks to different worker nodes? (Not the job scheduler)
> > Big thanks,
> > Shen
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: How to modify task assignment algorithm?
Posted by Jeff Zhang <zj...@gmail.com>.
I believe it is possible. But what is your purpose? I believe current
solution is good enough.
On Fri, Oct 8, 2010 at 2:57 AM, Shen LI <ge...@gmail.com> wrote:
> Hi,
> How can I modify the task assignment strategy in hadoop which is used to
> assign tasks to different worker nodes? (Not the job scheduler)
> Big thanks,
> Shen
--
Best Regards
Jeff Zhang