[jira] [Commented] (HIVE-21847) Reduce the communication rate between TezAM and LlapDaemons for Llap statistics

["Peter Vary (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-21847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900953#comment-16900953 ] 

Peter Vary commented on HIVE-21847:
-----------------------------------

Opening a new communication channel between TezAMs to query the data from the leader TezAM could be problematic, also it can put extra load on the leader TezAM adding unpredictable extra latency for the queries running on it.

Possible solutions:
 * New component to collect the data from LLAP Daemons, and TezAMs get the data from this new component - same new communication channels should be opened as for the original suggestion, but it is much more clear why/when this is needed.
 * If we do not want to use the metrics data for other things than the health check - then only the leader TezAM should query the data and act on it. Other (non-leader) TezAMs should only make sure that there is at least 1 leader TezAM

> Reduce the communication rate between TezAM and LlapDaemons for Llap statistics
> -------------------------------------------------------------------------------
>
>                 Key: HIVE-21847
>                 URL: https://issues.apache.org/jira/browse/HIVE-21847
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Peter Vary
>            Priority: Major
>
> HIVE-21846 is suboptimal if we have big number of TezAMs and LlapDaemons.
> We have to find a better way which is scaleable.
> Possible solution could be:
>  * Elect a TezAM leader for statistics fetching using ZooKeeper for the election
>  * TezAM should become a leader and fetch LlapDaemon data from the Daemons if there is no existing leader
>  * TezAM should query the data from the leader TezAM if there is an existing leader.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)