Which thing in Hama is the same like the "Slot" in hadoop?

[Shuo Wang <ec...@gmail.com>]

Hi,

Which thing in Hama is the same like the "*Slot*" in hadoop?

Best Wishes
Wang Shuo

Re: Which thing in Hama is the same like the "Slot" in hadoop?

[Suraj Menon <su...@apache.org>]

Hi Wang,

In Hama, every task has same category and priority. You would have already
observed that GroomServer acts as a container/observer for every BSP task.
Every GroomServer reports the status of itself and these tasks to
BSPMaster.  When GroomServer comes up, it is configured to run a particular
number of tasks on the machine('bsp.tasks.maximum' property mentioned by
Edward above) . So a scheduler has to decide the best choice of
GroomServers with the available slots to allocate the tasks in a job.

In the previous code refactor, we added
org.apache.hama.bsp.taskallocation.TaskAllocationStrategy.
You may modify this interface for the purpose. The idea is to make it
pluggable. Currently we have something that does a best effort attempt for
data locality. However, bsp solutions are heavy in messaging. Hence, we
have not yet handled the task group locality along with data locality; in
simple words, the tasks in a job should preferrably be scheduled in the
same rack instead of separate ones.

The other aspects where you want to define the resources (CPU, RAM,
(GPU/x86) ) per node, you can make them a property for GroomServer to
report to BSPMaster and the scheduler can decide to allocate resources
based on a global view of cluster. You may also look into YARN or Mesos
before proceeding.

Hope it helps :)

Thanks,
Suraj




On Wed, Apr 10, 2013 at 5:09 AM, Edward J. Yoon <ed...@apache.org>wrote:

> If I understand/remember 'Fair Scheduler' correctly, it adjusts the
> balance of the multiple MR jobs by killing tasks (changing to peding
> status).
>
> In Hama BSP case, I don't have any idea yet. Let's think about it together.
>
> On Wed, Apr 10, 2013 at 5:54 PM, Shuo Wang <ec...@gmail.com>
> wrote:
> > If I want to implement the fair scheduler in Hama, how can I allocate the
> > resources between different jobs.I know that hadoop use slot to represent
> > the resource, it actually allocates the slots.
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

Re: Which thing in Hama is the same like the "Slot" in hadoop?

["Edward J. Yoon" <ed...@apache.org>]

If I understand/remember 'Fair Scheduler' correctly, it adjusts the
balance of the multiple MR jobs by killing tasks (changing to peding
status).

In Hama BSP case, I don't have any idea yet. Let's think about it together.

On Wed, Apr 10, 2013 at 5:54 PM, Shuo Wang <ec...@gmail.com> wrote:
> If I want to implement the fair scheduler in Hama, how can I allocate the
> resources between different jobs.I know that hadoop use slot to represent
> the resource, it actually allocates the slots.



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: Which thing in Hama is the same like the "Slot" in hadoop?

[Shuo Wang <ec...@gmail.com>]

If I want to implement the fair scheduler in Hama, how can I allocate the
resources between different jobs.I know that hadoop use slot to represent
the resource, it actually allocates the slots.

Re: Which thing in Hama is the same like the "Slot" in hadoop?

["Edward J. Yoon" <ed...@apache.org>]

Long time no see, Wang Shuo, If you mean the slots for map or reduce
tasks, see 'bsp.tasks.maximum' property.

In Hadoop, Master always checks how many free slots are in each
TaskTracker to assign pending/failure tasks or new jobs (fair
scheduler). By the same token (node failures or new jobs), Hama also
should check the free slots in the future.

On Wed, Apr 10, 2013 at 5:17 PM, Shuo Wang <ec...@gmail.com> wrote:
> Hi,
>
> Which thing in Hama is the same like the "*Slot*" in hadoop?
>
> Best Wishes
> Wang Shuo



-- 
Best Regards, Edward J. Yoon
@eddieyoon