[jira] Commented: (MAPREDUCE-1044) Ability to automatically move machines from one MR compute cluster to another

["Arun C Murthy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780882#action_12780882 ] 

Arun C Murthy commented on MAPREDUCE-1044:
------------------------------------------

This proposal seems like we are about to re-invent Torque by adding yet another daemon to Hadoop Map-Reduce. You can also use HoD, along with features in Torque, to have a few large clusters and move tasktrackers around (via Torque). I do not have all the details, but I believe Torque can have custom monitoring and can be used to do smarter (i.e. Map-Reduce aware) scheduling.

If isolation really is the end-goal one can use full VMs right now. The capacity-scheduler, in conjunction with TaskController infrastructure in the TaskTracker, has some of the features you want: it does monitoring of memory consumed by the task process tree and ensures they do not go over a limit. Yes, it's harder to do cpu/io monitoring - but it is something everyone is looking to do. Your efforts in this space will be very useful to the whole community at-large... as indicated by our collaboration on MAPREDUCE-220 and other related jiras.

-1 for the direction proposed in this jira.

> Ability to automatically move machines from one MR compute cluster to another
> -----------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1044
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1044
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: DCD.pdf
>
>
> We have multiple map-reduce clusters that provide different service and support levels for their users. We have seen that utilization of hardware resources are not optimized if we have a static partition of existing hardware resources into these separate MR clusters. It would be nice to have a automatic way to move nodes from one MR cluster to another based on load characteristics and configured policies. This JIRA will discuss some of the ideas and possible implementations of those ideas.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.