[jira] [Created] (HUDI-1194) Reorganize HoodieHiveClient and make it fully support Hive Metastore API

["Wenning Ding (Jira)" <ji...@apache.org>]

Wenning Ding created HUDI-1194:
----------------------------------

             Summary: Reorganize HoodieHiveClient and make it fully support Hive Metastore API
                 Key: HUDI-1194
                 URL: https://issues.apache.org/jira/browse/HUDI-1194
             Project: Apache Hudi
          Issue Type: Improvement
            Reporter: Wenning Ding


Currently there are three ways in HoodieHiveClient to perform Hive functionalities. One is through Hive JDBC, one is through Hive Metastore API. One is through Hive Driver.
 
 There’s a parameter called +{{hoodie.datasource.hive_sync.use_jdbc}}+ to control whether use Hive JDBC or not. However, this parameter does not accurately describe the situation.


 Basically, current logic is when set +*use_jdbc*+ to true, most of the methods in HoodieHiveClient will use JDBC, and few methods in HoodieHiveClient will use Hive Metastore API.
 When set +*use_jdbc*+ to false, most of the methods in HoodieHiveClient will use Hive Driver, and few methods in HoodieHiveClient will use Hive Metastore API.

Here is a table shows that what will actually be used when setting use_jdbc to ture/false.
|Method|use_jdbc=true|use_jdbc=false|
|{{addPartitionsToTable}}|JDBC|Hive Driver|
|{{updatePartitionsToTable}}|JDBC|Hive Driver|
|{{scanTablePartitions}}|Metastore API|Metastore API|
|{{updateTableDefinition}}|JDBC|Hive Driver|
|{{createTable}}|JDBC|Hive Driver|
|{{getTableSchema}}|JDBC|Metastore API|
|{{doesTableExist}}|Metastore API|Metastore API|
|getLastCommitTimeSynced|Metastore API|Metastore API|

[~bschell] and I developed several Metastore API implementation for {{createTable, }}{{addPartitionsToTable}}{{, }}{{updatePartitionsToTable}}{{, }}{{updateTableDefinition }}{{which will be helpful for several issues: e.g. resolving null partition hive sync issue and supporting ALTER_TABLE cascade with AWS glue catalog}}{{. }}

{{But it seems hard to organize three implementations within the current config. So we plan to separate HoodieHiveClient into three classes:}}
 # {{HoodieHiveClient which implements all the APIs through Metastore API.}}
 # {{HoodieHiveJDBCClient which extends from HoodieHiveClient and overwrite several the APIs through Hive JDBC.}}
 # {{HoodieHiveDriverClient which extends from HoodieHiveClient and overwrite several the APIs through Hive Driver.}}

{{And we introduce a new parameter }}+*hoodie.datasource.hive_sync.hive_client_class*+ which could** _**_ let you choose which Hive Client class to use.

{{}}

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)