[jira] [Updated] (GOBBLIN-1398) Gobblin kerberos token management for multiple remote hive metastores

["Jay Sen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GOBBLIN-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Sen updated GOBBLIN-1398:
-----------------------------
    Description: 
GOBBLIN-1308 takes care of ability to connect to remote and secure hive metastore, but it still requires the management of hive-site to be provided to be container local file. 

When it comes to multiple hive clusters this manual management approach would not work, and requires special feature of providing system specific hive-site.xml without namespace collision.

 

This ticket aims to do following things

1) define a way to provide remote system configuration ( keep giving flat config is more cumbersome )

2) based on system config and feature flag, copy config files to container local path automatically.

3) when creating metastoreClient, pick up the right config for the requested system ( identified by the metastore URI )

 

 

  was:
Gobblin's hadoop tokens/ key management : 
 Problem: Gobblin only maintains local cluster tokens when key management is enabled. and does not have capability to manage tokens for remote hadoop cluster. ( based on my conversation with many folks here, the token files can be made available externally. but that would require that external system running on cron or something )

Solution: add remote cluster token management in Gobblin. where remote clusters key can be managed same way it manages the local clusters keys.

 

Config looks like following

( Changes the enable.key.management config to key.management.enabled )

 
{code:java}
gobblin.hadoop.key.management {
 enabled = true
 remote.clusters = [ ${gobblin_sync_systems.hadoop_cluster1}, ${gobblin_sync_systems.hadoop_cluster2} ]
}

// These Gobblin platform configurations can be moved to database for other use-cases, but this layout helps make the platform moduler for each connectors.
gobblin_sync_systems {
 hadoop_cluster1 {
 // if Hadoop config path is specified, the FileSystem will be created based on all the xml config provided here, which has all the required info.
 hadoop_config_path = "file:///etc/hadoop_cluster1/hadoop/config"
 // If hadoop config path is not specified, you can still specify the speecific nodes for the specific type of tokens
 namenode_uri = ["hdfs://nn1.hadoop_cluster1.example.com:8020", "hdfs://nn2.hadoop_cluster1.example.com:8020"]
 kms_nodes = [ "kms1.hadoop_cluster1.example.com:9292", "kms2.hadoop_cluster1.example.com:9292" ]
 }
 hadoop_cluster2 {
 hadoop_config_path = "file:///etc/hadoop_cluster1/hadoop/config"
 namenode_uri = ["hdfs://nn1.hadoop_cluster2.example.com:8020", "hdfs://nn2.hadoop_cluster2.example.com:8020"]
 kms_nodes = [ "kms1.hadoop_cluster2.example.com:9292", "kms2.hadoop_cluster2.example.com:9292" ]
 }
}{code}


> Gobblin kerberos token management for multiple remote hive metastores
> ---------------------------------------------------------------------
>
>                 Key: GOBBLIN-1398
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1398
>             Project: Apache Gobblin
>          Issue Type: Improvement
>    Affects Versions: 0.15.0
>            Reporter: Jay Sen
>            Priority: Major
>             Fix For: 0.16.0
>
>
> GOBBLIN-1308 takes care of ability to connect to remote and secure hive metastore, but it still requires the management of hive-site to be provided to be container local file. 
> When it comes to multiple hive clusters this manual management approach would not work, and requires special feature of providing system specific hive-site.xml without namespace collision.
>  
> This ticket aims to do following things
> 1) define a way to provide remote system configuration ( keep giving flat config is more cumbersome )
> 2) based on system config and feature flag, copy config files to container local path automatically.
> 3) when creating metastoreClient, pick up the right config for the requested system ( identified by the metastore URI )
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)