[jira] [Resolved] (IMPALA-1032) SimpleScheduler::ComputeScanRangeAssignment doesn't load balance across non-HDFS range

["Lars Volker (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars Volker resolved IMPALA-1032.
---------------------------------
    Resolution: Fixed

commit 0ad935b63c23029bd4cac4beefde6b3b7c0e322b
Author: Lars Volker <lv...@cloudera.com>
Date:   Fri Feb 12 16:31:36 2016 -0800

    IMPALA-2979: Fix scheduling on remote hosts

    Also fixes: IMPALA-2400, IMPALA-3043

    This change fixes scheduling scan-ranges on remote hosts by adding
    remote backend selection capability to SimpleScheduler. Prior to this
    change the scheduler would try to select a local backend even when
    remote scheduling was requested.

    This change also allows pseudo-randomized remote backend selection to
    prevent convoying, which could happen when different independent
    schedulers had the same internal state, e.g. after a cluster restart. To
    enable the new behavior set the query option SCHEDULE_RANDOM_REPLICA to
    true.

    This change also fixes IMPALA-2400: Unpredictable locality behavior
    for reading Parquet files

    This change also fixes IMPALA-3043: SimpleScheduler does not handle
    hosts with multiple IP addresses correctly

    This change also does some clean-up in scheduler.h and
    simple-scheduler.{h,cc}.

    Change-Id: I044f83806fcde820fcb38047cf6b8e780d803858
    Reviewed-on: http://gerrit.cloudera.org:8080/3771
    Reviewed-by: Lars Volker <lv...@cloudera.com>
    Reviewed-by: Sailesh Mukil <sa...@cloudera.com>
    Tested-by: Internal Jenkins

> SimpleScheduler::ComputeScanRangeAssignment doesn't load balance across non-HDFS range
> --------------------------------------------------------------------------------------
>
>                 Key: IMPALA-1032
>                 URL: https://issues.apache.org/jira/browse/IMPALA-1032
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 1.3.1
>            Reporter: Todd Lipcon
>            Priority: Minor
>
> In ComputeScanRangeAssignment, we set the scan_range_length to 0 if the scan range is not an HDFS file split. This has the effect of not balancing across scan range locations if multiple locations are provided -- all fragments are assigned to the first replica. Setting to an arbitrary integer like 1000 seems to solve the issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)