[jira] [Commented] (HIVE-14822) Add support for credential provider for jobs launched from Hiveserver2

["Hive QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530873#comment-15530873 ] 

Hive QA commented on HIVE-14822:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830598/HIVE-14822.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 10651 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_stats_orc]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_excludeHadoop20]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_8]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fetch_aggregation]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonmr_fetch]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_analyze]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample10]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample_islocalmode_hook]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample_islocalmode_hook_use_metadata]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_parquet]
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[dboutput]
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testCliDriver[case_with_row_sequence]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan1
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan2
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan1
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan2
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan4
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan5
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan6
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1332/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1332/console
Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1332/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12830598 - PreCommit-HIVE-Build

> Add support for credential provider for jobs launched from Hiveserver2
> ----------------------------------------------------------------------
>
>                 Key: HIVE-14822
>                 URL: https://issues.apache.org/jira/browse/HIVE-14822
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>            Reporter: Vihang Karajgaonkar
>            Assignee: Vihang Karajgaonkar
>         Attachments: HIVE-14822.01.patch
>
>
> When using encrypted passwords via the Hadoop Credential Provider, HiveServer2 currently does not correctly forward enough information to the job configuration for jobs to read those secrets. If your job needs to access any secrets, like S3 credentials, then there's no convenient and secure way to configure this today.
> You could specify the decryption key in files like mapred-site.xml that HiveServer2 uses, but this would place the encryption password on local disk in plaintext, which can be a security concern.
> To solve this problem, HiveServer2 should modify job configuration to include the environment variable settings needed to decrypt the passwords. Specifically, it will need to modify:
> * For MR2 jobs:
> ** yarn.app.mapreduce.am.admin.user.env
> ** mapreduce.admin.user.env
> * For Spark jobs:
> ** spark.yarn.appMasterEnv.HADOOP_CREDSTORE_PASSWORD
> ** spark.executorEnv.HADOOP_CREDSTORE_PASSWORD
> HiveServer2 can get the decryption password from its own environment, the same way it does for its own credential provider store today.
> Additionally, it can be desirable for HiveServer2 to have a separate encrypted password file than what is used by the job. HiveServer2 may have secrets that the job should not have, such as the metastore database password or the password to decrypt its private SSL certificate. It is also best practices to have separate passwords on separate files. To facilitate this, Hive will also accept:
> * A configuration for a path to a credential store to use for jobs. This should already be uploaded in HDFS. (hive.server2.job.keystore.location or a better name) If this is not specified, then HS2 will simply use the value of hadoop.security.credential.provider.path.
> * An environment variable for the password to decrypt the credential store (HIVE_JOB_KEYSTORE_PASSWORD or better). If this is not specified, then HS2 will simply use the standard environment variable for decrypting the Hadoop Credential Provider.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)