You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "kevin (JIRA)" <ji...@apache.org> on 2018/09/07 10:32:00 UTC
[jira] [Updated] (OOZIE-3333) error when running hiveserver2 behind
the load balancer, When running through beeline it is working fine but
getting error when running through oozie
[ https://issues.apache.org/jira/browse/OOZIE-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
kevin updated OOZIE-3333:
-------------------------
Description:
{noformat}
[HiveServer2-Handler-Pool: Thread-198]: SASL negotiation failure
javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: token expired or does not exist: HIVE_DELEGATION_TOKEN owner=hadoop, renewer=hive, realUser=hive/bdpnode2.dev.com@HADOOP.COM, issueDate=1535678531257, maxDate=1536283331257, sequenceNumber=18, masterKeyId=4]
at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:793)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:790)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:790)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: token expired or does not exist: HIVE_DELEGATION_TOKEN owner=hadoop, renewer=hive, realUser=hive/bdpnode2.dev.com@HADOOP.COM, issueDate=1535678531257, maxDate=1536283331257, sequenceNumber=18, masterKeyId=4
at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:114)
at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:56)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.getPassword(HadoopThriftAuthBridge.java:607)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.handle(HadoopThriftAuthBridge.java:638)
at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
... 15 more
8月 31, 上午9点22:25.582分 ERROR org.apache.thrift.server.TThreadPoolServer
[HiveServer2-Handler-Pool: Thread-198]: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: DIGEST-MD5: IO error acquiring password
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:793)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:790)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:790)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: DIGEST-MD5: IO error acquiring password
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
kerberos-secured cluster
error when running more hiveserver2 behind the load balancer
,but a single hiverserver2 works fine
oozie hiveserver2 script -> haproxy -> hiveserver2(more)
job.properties:
oozie.use.system.libpath=True
send_email=False
dryrun=False
credentials={u'hcat':
{'xml_name': u'hcat', 'properties': [('hcat.metastore.uri', u'thrift://bdpnode1.dev.com:9083'), ('hcat.metastore.principal', u'hive/bdpnode1.dev.com@HADOOP.COM')]}
, u'hive2':
{'xml_name': u'hive2', 'properties': [('hive2.jdbc.url', 'jdbc:hive2://bdpnode2.dev.com:10001/default'), ('hive2.server.principal', u'hive/bdpnode2.dev.com@HADOOP.COM')]}
, u'hbase': {'xml_name': u'hbase', 'properties': []}}
nameNode=hdfs://ns2
jobTracker=yarnRM
security_enabled=True
workflow.xml:
<workflow-app name="hivetest" xmlns="uri:oozie:workflow:0.5">
<credentials>
<credential name="hcat" type="hcat">
<property>
<name>hcat.metastore.uri</name>
<value>thrift://bdpnode1.dev.com:9083</value>
</property>
<property>
<name>hcat.metastore.principal</name>
<value>hive/bdpnode1.dev.com@HADOOP.COM</value>
</property>
</credential>
<credential name="hive2" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>jdbc:hive2://bdpnode2.dev.com:10001/default</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>hive/bdpnode2.dev.com@HADOOP.COM</value>
</property>
</credential>
</credentials>
<start to="hive2-dff4"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="hive2-dff4" cred="hive2,hcat">
<hive2 xmlns="uri:oozie:hive2-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<jdbc-url>jdbc:hive2://bdpnode2.dev.com:10001/default</jdbc-url>
<script>/user/hadoop/wf/hive.q</script>
</hive2>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>
{noformat}
was:
{noformat}
[HiveServer2-Handler-Pool: Thread-198]: SASL negotiation failure
javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: token expired or does not exist: HIVE_DELEGATION_TOKEN owner=hadoop, renewer=hive, realUser=hive/bdpnode2.dev.com@HADOOP.COM, issueDate=1535678531257, maxDate=1536283331257, sequenceNumber=18, masterKeyId=4]
at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:793)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:790)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:790)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: token expired or does not exist: HIVE_DELEGATION_TOKEN owner=hadoop, renewer=hive, realUser=hive/bdpnode2.dev.com@HADOOP.COM, issueDate=1535678531257, maxDate=1536283331257, sequenceNumber=18, masterKeyId=4
at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:114)
at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:56)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.getPassword(HadoopThriftAuthBridge.java:607)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.handle(HadoopThriftAuthBridge.java:638)
at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
... 15 more
8月 31, 上午9点22:25.582分 ERROR org.apache.thrift.server.TThreadPoolServer
[HiveServer2-Handler-Pool: Thread-198]: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: DIGEST-MD5: IO error acquiring password
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:793)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:790)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:790)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: DIGEST-MD5: IO error acquiring password
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
{noformat}
> error when running hiveserver2 behind the load balancer, When running through beeline it is working fine but getting error when running through oozie
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: OOZIE-3333
> URL: https://issues.apache.org/jira/browse/OOZIE-3333
> Project: Oozie
> Issue Type: Bug
> Components: action
> Affects Versions: 4.3.0
> Reporter: kevin
> Priority: Blocker
>
> {noformat}
> [HiveServer2-Handler-Pool: Thread-198]: SASL negotiation failure
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: token expired or does not exist: HIVE_DELEGATION_TOKEN owner=hadoop, renewer=hive, realUser=hive/bdpnode2.dev.com@HADOOP.COM, issueDate=1535678531257, maxDate=1536283331257, sequenceNumber=18, masterKeyId=4]
> at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
> at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
> at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
> at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:793)
> at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:790)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
> at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:790)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: token expired or does not exist: HIVE_DELEGATION_TOKEN owner=hadoop, renewer=hive, realUser=hive/bdpnode2.dev.com@HADOOP.COM, issueDate=1535678531257, maxDate=1536283331257, sequenceNumber=18, masterKeyId=4
> at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:114)
> at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:56)
> at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.getPassword(HadoopThriftAuthBridge.java:607)
> at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.handle(HadoopThriftAuthBridge.java:638)
> at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
> ... 15 more
> 8月 31, 上午9点22:25.582分 ERROR org.apache.thrift.server.TThreadPoolServer
> [HiveServer2-Handler-Pool: Thread-198]: Error occurred during processing of message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: DIGEST-MD5: IO error acquiring password
> at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:793)
> at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:790)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
> at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:790)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.transport.TTransportException: DIGEST-MD5: IO error acquiring password
> at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
> at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> ... 10 more
> kerberos-secured cluster
> error when running more hiveserver2 behind the load balancer
> ,but a single hiverserver2 works fine
>
> oozie hiveserver2 script -> haproxy -> hiveserver2(more)
> job.properties:
> oozie.use.system.libpath=True
> send_email=False
> dryrun=False
> credentials={u'hcat':
> {'xml_name': u'hcat', 'properties': [('hcat.metastore.uri', u'thrift://bdpnode1.dev.com:9083'), ('hcat.metastore.principal', u'hive/bdpnode1.dev.com@HADOOP.COM')]}
> , u'hive2':
> {'xml_name': u'hive2', 'properties': [('hive2.jdbc.url', 'jdbc:hive2://bdpnode2.dev.com:10001/default'), ('hive2.server.principal', u'hive/bdpnode2.dev.com@HADOOP.COM')]}
> , u'hbase': {'xml_name': u'hbase', 'properties': []}}
> nameNode=hdfs://ns2
> jobTracker=yarnRM
> security_enabled=True
> workflow.xml:
> <workflow-app name="hivetest" xmlns="uri:oozie:workflow:0.5">
> <credentials>
> <credential name="hcat" type="hcat">
> <property>
> <name>hcat.metastore.uri</name>
> <value>thrift://bdpnode1.dev.com:9083</value>
> </property>
> <property>
> <name>hcat.metastore.principal</name>
> <value>hive/bdpnode1.dev.com@HADOOP.COM</value>
> </property>
> </credential>
> <credential name="hive2" type="hive2">
> <property>
> <name>hive2.jdbc.url</name>
> <value>jdbc:hive2://bdpnode2.dev.com:10001/default</value>
> </property>
> <property>
> <name>hive2.server.principal</name>
> <value>hive/bdpnode2.dev.com@HADOOP.COM</value>
> </property>
> </credential>
> </credentials>
> <start to="hive2-dff4"/>
> <kill name="Kill">
> <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <action name="hive2-dff4" cred="hive2,hcat">
> <hive2 xmlns="uri:oozie:hive2-action:0.1">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <jdbc-url>jdbc:hive2://bdpnode2.dev.com:10001/default</jdbc-url>
> <script>/user/hadoop/wf/hive.q</script>
> </hive2>
> <ok to="End"/>
> <error to="Kill"/>
> </action>
> <end name="End"/>
> </workflow-app>
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)