[jira] [Resolved] (PHOENIX-6694) Avoid unnecessary calls of fetching table meta data to region servers holding the system tables in batch oriented jobs in spark or hive otherwise those RS become hotspot

["Rajeshbabu Chintaguntla (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PHOENIX-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rajeshbabu Chintaguntla resolved PHOENIX-6694.
----------------------------------------------
    Resolution: Fixed

committed to master branch. Thanks for the reviews [~stoty],[~elserj].

> Avoid unnecessary calls of fetching table meta data to region servers holding the system tables in batch oriented jobs in spark or hive otherwise those RS become hotspot
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-6694
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-6694
>             Project: Phoenix
>          Issue Type: Task
>          Components: hive-connector, spark-connector
>            Reporter: Rajeshbabu Chintaguntla
>            Assignee: Rajeshbabu Chintaguntla
>            Priority: Major
>
> Currently we are preparing the query plan in both data source and partition readers which is creating new connection in each worker and job initialisation  which unnecessarily  touch basing all both system catalog table, system stats table as well as meta. When there are jobs with millions of parallel workers hotspot the region servers holding the meta and system catalog as well system stats table. So if we share the same query plan between the workers which can avoid the hotspot.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)