[jira] [Commented] (HIVE-11265) LLAP: investigate locality issues

["Sergey Shelukhin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-11265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15011836#comment-15011836 ] 

Sergey Shelukhin commented on HIVE-11265:
-----------------------------------------

Anything more to do here? There was some investigation, it might be valuable to record the results and what needs to be done if anything [~sseth]

> LLAP: investigate locality issues
> ---------------------------------
>
>                 Key: HIVE-11265
>                 URL: https://issues.apache.org/jira/browse/HIVE-11265
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sergey Shelukhin
>            Assignee: Siddharth Seth
>
> Running q27 with split-waves 0.9 on 10 nodes x 16 executors, I get 140 mappers reading store_sales, and 5~ more assorted vertices.
> When running the query repeatedly, one would expect good locality, i.e. the same splits (files+stripes) being processed on the same nodes most of the time.
> However, this is only the case for 40-50% of the stripes in my experience. When the query is run 10 times in a row, an average split (file+stripe) is read on ~4 machines. Some are actually read on a different machine every run :)
> This affects cache hit ratio.
> Understandably in real scenarios we won't get 100% locality, but we should not be getting bad locality in simple cases like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)