You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Hive QA (JIRA)" <ji...@apache.org> on 2017/09/02 05:40:00 UTC
[jira] [Commented] (HIVE-17368) DBTokenStore fails to connect in
Kerberos enabled remote HMS environment
[ https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151416#comment-16151416 ]
Hive QA commented on HIVE-17368:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12885052/HIVE-17368.03-branch-2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6654/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6654/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6654/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-09-02 05:39:27.367
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-6654/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z branch-2 ]]
+ [[ -d apache-github-branch-2-source ]]
+ [[ ! -d apache-github-branch-2-source/.git ]]
+ [[ ! -d apache-github-branch-2-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-09-02 05:39:27.370
+ cd apache-github-branch-2-source
+ git fetch origin
From https://github.com/apache/hive
588148d..76933e7 branch-2 -> origin/branch-2
5a62503..714d7cf branch-2.1 -> origin/branch-2.1
120476d..b2e7d5e branch-2.2 -> origin/branch-2.2
6f4c35c..dee0a20 branch-2.3 -> origin/branch-2.3
6be50b7..d155565 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 588148d HIVE-17327 : ADDENDUM (revert a small part of the patch to fix the test) (Sergey Shelukhin)
+ git clean -f -d
+ git checkout branch-2
Already on 'branch-2'
Your branch is behind 'origin/branch-2' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/branch-2
HEAD is now at 76933e7 HIVE-17411 : LLAP IO may incorrectly release a refcount in some rare cases (Sergey Shelukhin, reviewed by Prasanth Jayachandran)
+ git merge --ff-only origin/branch-2
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-09-02 05:39:33.815
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p1
patching file itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/MiniHiveKdc.java
patching file itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithDBTokenStore.java
patching file itests/hive-unit-hadoop2/src/test/java/org/apache/hadoop/hive/thrift/TestHadoopAuthBridge23.java
patching file itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java
patching file ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
patching file service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
patching file shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java
patching file shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java
patching file shims/common/src/main/java/org/apache/hadoop/hive/thrift/HiveDelegationTokenManager.java
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] COMPILATION ERROR :
[ERROR] /data/hiveptest/working/apache-github-branch-2-source/shims/common/src/main/java/org/apache/hadoop/hive/thrift/HiveDelegationTokenManager.java:[124,25] cannot find symbol
symbol: method getDSeelegationToken(java.lang.String,java.lang.String)
location: variable secretManager of type org.apache.hadoop.hive.thrift.DelegationTokenSecretManager
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-shims-common: Compilation failure
[ERROR] /data/hiveptest/working/apache-github-branch-2-source/shims/common/src/main/java/org/apache/hadoop/hive/thrift/HiveDelegationTokenManager.java:[124,25] cannot find symbol
[ERROR] symbol: method getDSeelegationToken(java.lang.String,java.lang.String)
[ERROR] location: variable secretManager of type org.apache.hadoop.hive.thrift.DelegationTokenSecretManager
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hive-shims-common
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12885052 - PreCommit-HIVE-Build
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
> ------------------------------------------------------------------------
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
> Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
> Reporter: Vihang Karajgaonkar
> Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch, HIVE-17368.02-branch-2.patch, HIVE-17368.03-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos, and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not able to invoke HMS APIs needed to add/remove/renew tokens from the DB since it is possible that the user which is issue the {{GetDelegationToken}} is not kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This principal can establish a transport authenticated using Kerberos. It stores the HMS delegation token string in the sessionConf and sessionToken. Now, lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call cannot instantiate a HMSClient and open transport to HMS using the HMSToken string available in the sessionConf, since DBTokenStore uses server HiveConf instead of sessionConf. It tries to establish transport using Kerberos and it fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61] transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[?:1.8.0_121]
> at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) ~[libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) [libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) [libthrift-0.9.3.jar:0.9.3]
> at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) [hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488) [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:255) [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70) [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0_121]
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0_121]
> at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699) [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83) [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3595) [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3647) [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3627) [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:157) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.thrift.DBTokenStore.addToken(DBTokenStore.java:74) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.createPassword(TokenStoreDelegationTokenSecretManager.java:142) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.createPassword(TokenStoreDelegationTokenSecretManager.java:56) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.security.token.Token.<init>(Token.java:59) [hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.hive.thrift.DelegationTokenSecretManager.getDelegationToken(DelegationTokenSecretManager.java:109) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager$1.run(HiveDelegationTokenManager.java:123) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager$1.run(HiveDelegationTokenManager.java:119) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) [hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.getDelegationToken(HiveDelegationTokenManager.java:119) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.getDelegationTokenWithService(HiveDelegationTokenManager.java:130) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hive.service.auth.HiveAuthFactory.getDelegationToken(HiveAuthFactory.java:261) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.getDelegationToken(HiveSessionImplwithUGI.java:174) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) [hadoop-common-2.7.2.jar:?]
> at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at com.sun.proxy.$Proxy36.getDelegationToken(Unknown Source) [?:?]
> at org.apache.hive.service.cli.CLIService.getDelegationToken(CLIService.java:589) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hive.service.cli.thrift.ThriftCLIService.GetDelegationToken(ThriftCLIService.java:254) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetDelegationToken.getResult(TCLIService.java:1737) [hive-service-rpc-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetDelegationToken.getResult(TCLIService.java:1722) [hive-service-rpc-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [libthrift-0.9.3.jar:0.9.3]
> at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:621) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [libthrift-0.9.3.jar:0.9.3]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
> Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[?:1.8.0_121]
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) ~[?:1.8.0_121]
> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[?:1.8.0_121]
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) ~[?:1.8.0_121]
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[?:1.8.0_121]
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_121]
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_121]
> ... 65 more
> {noformat}
> On HMS side I see a exception saying
> {noformat}
> 2017-08-17 11:45:13,655 ERROR org.apache.thrift.server.TThreadPoolServer: [pool-7-thread-34]: Error occurred during processing of message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: DIGEST-MD5: IO error acquiring password
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)