[jira] [Commented] (TAJO-743) Change the default resource allocation policy of leaf tasks

["Jinho Kim (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TAJO-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963899#comment-13963899 ] 

Jinho Kim commented on TAJO-743:
--------------------------------

Created a review request against branch master in reviewboard
https://reviews.apache.org/r/20149/


> Change the default resource allocation policy of leaf tasks
> -----------------------------------------------------------
>
>                 Key: TAJO-743
>                 URL: https://issues.apache.org/jira/browse/TAJO-743
>             Project: Tajo
>          Issue Type: Improvement
>          Components: resource manager
>    Affects Versions: 0.8-incubating, 1.0-incubating
>            Reporter: Jinho Kim
>            Assignee: Jinho Kim
>         Attachments: TAJO-743.patch, TAJO-743_branch-0.8.0.patch
>
>
> Currently, resource allocation is calculated by memory base. If a machine have a large memory, in default settings, heavy disk IO per disk is usually caused by high task concurrency. However, it is likely to seem to be problematic.
> When i tested the leaf task scan by 2(concurrency of SATA disk), the performance was better. if you have SAS Storage or SSD, you can increase the disk concurrency. This patch changes the default resource allocation policy to use disk resource.
> The following configs have been available so far:
>  * tajo.worker.resource.disks - available disk resource of each worker
>  * tajo.task.disk-slot.default - how many disk resource is consumed per task
> Below config is newly introduced in this patch
>  * tajo.worker.resource.dfs-dir-aware - it can be true/false. If it is true, each worker uses the number of HDFS datanode's data dirs in the worker as the disk resource. So, tajo.worker.resource.disks is ignored.



--
This message was sent by Atlassian JIRA
(v6.2#6252)