[jira] [Comment Edited] (TAJO-317) Improve TajoResourceManager to support more elaborate resource management

["Keuntae Park (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TAJO-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836222#comment-13836222 ] 

Keuntae Park edited comment on TAJO-317 at 12/2/13 4:11 AM:
------------------------------------------------------------

Sorry for the late answer, Jihoon and Hyunsik.

I've uploaded new patch that reflects all your comments:
- when dedicated=false, use default values (memory size is 512M, number of disk is 1.0)
- use systemConf.getIntVar() instead of systemConf.getInt()

I really thank you guys for the review,
and please let me know what I miss yet.


was (Author: sirpkt):
Sorry for the late answer, Jihoon and Hyunsik.

I've uploaded new patch that reflects all your comments:
- when dedicated=false, use default values (memory size is 512M, number of disk is 1.0)
- use systemConf.getIntVar() instead of systemConf.getInt()

I really thank you guys for the review,
and please let me know that I miss yet.

> Improve TajoResourceManager to support more elaborate resource management
> -------------------------------------------------------------------------
>
>                 Key: TAJO-317
>                 URL: https://issues.apache.org/jira/browse/TAJO-317
>             Project: Tajo
>          Issue Type: Improvement
>          Components: resource manager
>            Reporter: Hyunsik Choi
>            Assignee: Keuntae Park
>             Fix For: 0.8-incubating
>
>         Attachments: TAJO-317.patch, TAJO-317_2.patch, TAJO-317_3.patch, TAJO-317_4.patch
>
>
> h3. Status of the current Tajo Resource Manager (RM)
>  * Tajo RM manages CPU, DISK resource incompletely, and it only provides resource management through memory allocations. 
>  * In addition, Tajo RM considers the memory resource as the fixed number of slots.
> h3. Problem
> In many cases, workloads can be categorized into I/O intensive job and CPU and memory consuming job. For example, scan and hash partition or INSERT OVERWRITE may be belong to I/O intensive job. In general, Aggregation can be belong to CPU-memory consuming job. The current RM is not fit to support selectively I/O intensive job or CPU-memory consuming job because it provides only memory slots. We need more elaborate resource management mechanism.
> In addition, in most resource management systems, the remain resource less than required resource is not allocated in response to a resource request. It is not good to fully utilize the cluster resources. In order to mitigate this problem, we need to add resilience to allocation mechanism. For example, min-max request would be useful for it.
> h3. Proposal
>  * Tajo RM should provides resource management for disk and cpu-memory.
>  ** Tajo RM should provide allocation request call with min, max memory request, and min, max disk request.
>  *** min-max request will be useful to fully utilize remain cluster resources.
>  * Each resource request should have a priority. The priority can be disk or memory.
>   ** If the priority is disk
>   *** disk allocation will be limited depending on the remain disk resource
>   *** memory allocation will be not limited regardless of the remain memory resource, and just reduce the remain memory resource.
>   ** If the priority is memory
>    *** memory allocation will be limited depending on the remain memory resource
>    *** disk allocation will be not limited regardless of the remain disk resource, and just reduce the remain disk resource.
>  * disk resource in each worker is represented as a float value.
>  ** The initial disk resource will be the number of disks which participate in HDFS data directory.



--
This message was sent by Atlassian JIRA
(v6.1#6144)