[jira] [Comment Edited] (HIVE-25929) Let secret config properties to be propagated to Tez

["Alessandro Solimando (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-25929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489083#comment-17489083 ] 

Alessandro Solimando edited comment on HIVE-25929 at 2/9/22, 5:11 PM:
----------------------------------------------------------------------

I checked HIVE-9013 and HIVE-10508, the first one covers the "set $conf" (which basically only needs to be handled in SetProcessor, if I got it right), the latter strips some config properties from the configuration before sending it to the workers.

We can keep HIVE-9013 pretty easily, and I agree that we have two options for HIVE-10508:
# add an additional property "hive.conf.hidden.list.exec.engines"
# revert the patch and use "stripHivePasswordDetails", so we allow the storage credentials to reach the executors

For the first option, I'd rather set it to "fs.azure.account.oauth2.client.secret" + the S3 counterpart, and use "hive.conf.hidden.list" in SetProcessor and remove ("hive.conf.hidden.list" minus "fs.azure.account.oauth2.client.secret") from the config sent to the executors (by changing "stripHiddenConfigurations").

WDYT?


was (Author: asolimando):
I checked HIVE-9013 and HIVE-10508, the first one covers the "set $conf" (which basically only needs to be handled in SetProcessor, if I got it right), the latter strips some config properties from the configuration before sending it to the workers.

We can keep HIVE-9013 pretty easily, and I agree that we have two options for HIVE-10508:
# add an additional property "hive.conf.hidden.list.exec.engines"
# revert the patch and let the full config to reach the executors

For the first option, I'd rather set it to "fs.azure.account.oauth2.client.secret" + the S3 counterpart, and use "hive.conf.hidden.list" in SetProcessor and remove ("hive.conf.hidden.list" minus "fs.azure.account.oauth2.client.secret") from the config sent to the executors (by changing "stripHiddenConfigurations").

WDYT?

> Let secret config properties to be propagated to Tez
> ----------------------------------------------------
>
>                 Key: HIVE-25929
>                 URL: https://issues.apache.org/jira/browse/HIVE-25929
>             Project: Hive
>          Issue Type: Bug
>            Reporter: László Bodor
>            Assignee: László Bodor
>            Priority: Major
>
> History in chronological order:
> HIVE-10508: removed some passwords from config that's propagated to execution engines
> HIVE-9013: introduced hive.conf.hidden.list, which is used instead of the hardcoded list in HIVE-10508
> the problem with HIVE-9013 is it's about to introduce a common method for removing sensitive data from Configuration, which absolutely makes sense in most of the cases (set command showing sensitive data), but can cause issues e.g. while using non-secure cloud connectors on a cluster, where instead of the hadoop credential provider API (which is considered the secure and proper way), passwords/secrets appear in the Configuration object (like: "fs.azure.account.oauth2.client.secret")
> 2 possible solutions:
> 1. introduce a new property like: "hive.conf.hidden.list.exec.engines" -> which defaults to "hive.conf.hidden.list" (configurable, but maybe just more confusing to users, having a new config property which should be understood and maintained on a cluster)
> 2. simply revert DAGUtils to use to old stripHivePasswordDetails introduced by HIVE-10508 (convenient, less confusing for users, but cannot be configured)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)