You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by Gagan Brahmi <ga...@gmail.com> on 2017/03/23 18:15:25 UTC
HAWQ with Ranger KMS
Hi All,
Is there any known issue around running HAWQ in a cluster with Ranger KMS
over YARN?
Seems that HAWQ is not able to obtain containers when requesting it to YARN
ResourceManager.
The following is what I am seeing in the YARN RM logs:
---------------------
2017-03-23 10:56:30,816 INFO hdfs.DFSClient
(DFSClient.java:getDelegationToken(1043)) - Created HDFS_DELEGATION_TOKEN
token 20049 for postgres on 192.168.59.104:8020
2017-03-23 10:56:30,889 WARN security.DelegationTokenRenewer
(DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(895)) - Unable
to add the application to the delegation token renewer.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1032)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
at
org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2298)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:685)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.obtainSystemTokensForUser(DelegationTokenRenewer.java:679)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.requestNewHdfsDelegationToken(DelegationTokenRenewer.java:643)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:488)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:77)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:891)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:868)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1014)
... 16 more
Caused by:
org.apache.hadoop.security.authentication.client.AuthenticationException:
Authentication failed, URL:
http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres&user.name=yarn,
status: 403, message: Forbidden
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:278)
at
org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:170)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1019)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1014)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
... 17 more
---------------------
The following is what I see in Ranger KMS log (kms.lo)
---------------------
2017-03-23 11:02:00,734 DEBUG LimitLatch - Counting
up[http-bio-9292-Acceptor-0] latch=7
2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [uriBC] has
value [/kms/v1/]
2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [semicolon] has
value [-1]
2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [enc] has value
[ISO-8859-1]
2017-03-23 11:02:00,738 DEBUG AuthenticatorBase - Security checking request
OPTIONS /kms/v1/
2017-03-23 11:02:00,738 DEBUG RealmBase - No applicable constraints
defined
2017-03-23 11:02:00,738 DEBUG AuthenticatorBase - Not subject to any
constraint
2017-03-23 11:02:00,738 TRACE StandardWrapper - Returning non-STM instance
2017-03-23 11:02:00,739 DEBUG Http11Protocol - Socket:
[org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/
192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State
out: [OPEN]
2017-03-23 11:02:00,758 DEBUG Http11Processor - Error parsing HTTP request
header
java.io.EOFException: Unexpected EOF read on the socket
at
org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:169)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:990)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
2017-03-23 11:02:00,758 DEBUG Http11Protocol - Socket:
[org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/
192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State
out: [CLOSE
D]
2017-03-23 11:02:00,758 TRACE JIoEndpoint - Closing
socket:org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/
192.168.59.104,port=58547,localport=9292]
---------------------
The following is from the Ranger KMS access log:
---------------------
2017-03-23 11:02:00,738 UNAUTHENTICATED RemoteHost:192.168.59.104
Method:OPTIONS URL:
http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos
tgres ErrorMsg:'Authentication required'
2017-03-23 11:02:00,786 UNAUTHENTICATED RemoteHost:192.168.59.104
Method:OPTIONS URL:
http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos
tgres&user.name=yarn ErrorMsg:'Authentication required'
---------------------
The following is from the Ranger KMS audit log (kms-audit.log)
---------------------
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS
/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%
40gagan.com&doAs=postgres HTTP/1.1" 401 997
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS
/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%
40gagan.com&doAs=postgres HTTP/1.1" 403 258
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS
/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%
40gagan.com&doAs=postgres&user.name=yarn HTTP/1.1" 401 997
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS
/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%
40gagan.com&doAs=postgres&user.name=yarn HTTP/1.1" 403 258
---------------------
I have added the following proxyuser configuration in Ranger KMS as well:
hadoop.kms.proxyuser.postgres.users=*
hadoop.kms.proxyuser.postgres.hosts=*
hadoop.kms.proxyuser.yarn.users=*
hadoop.kms.proxyuser.yarn.hosts=*
The core-site.xml has the required proxyuser configuration as well:
hadoop.proxyuser.postgres.groups=*
hadoop.proxyuser.postgres.hosts=*
hadoop.proxyuser.yarn.groups=*
hadoop.proxyuser.yarn.hosts=*
But nothing seem to be working in this case here.
I would appreciate some inputs on this one.
Regards,
Gagan Brahmi
Re: HAWQ with Ranger KMS
Posted by Gagan Brahmi <ga...@gmail.com>.
This worked for me finally after Vineet's suggestion via HCC.
https://community.hortonworks.com/questions/90662/hawq-issues-with-ranger-kms.html
Thank you all.
Regards,
Gagan Brahmi
On Thu, Mar 23, 2017 at 11:15 AM, Gagan Brahmi <ga...@gmail.com>
wrote:
> Hi All,
>
> Is there any known issue around running HAWQ in a cluster with Ranger KMS
> over YARN?
>
> Seems that HAWQ is not able to obtain containers when requesting it to
> YARN ResourceManager.
>
> The following is what I am seeing in the YARN RM logs:
>
> ---------------------
>
> 2017-03-23 10:56:30,816 INFO hdfs.DFSClient (DFSClient.java:getDelegationToken(1043))
> - Created HDFS_DELEGATION_TOKEN token 20049 for postgres on
> 192.168.59.104:8020
> 2017-03-23 10:56:30,889 WARN security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(895)) - Unable
> to add the application to the delegation token renewer.
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.crypto.key.kms.KMSClientProvider.
> addDelegationTokens(KMSClientProvider.java:1032)
> at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExte
> nsion.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
> at org.apache.hadoop.hdfs.DistributedFileSystem.
> addDelegationTokens(DistributedFileSystem.java:2298)
> at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:685)
> at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1724)
> at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer.obtainSystemTokensForUser(
> DelegationTokenRenewer.java:679)
> at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer.requestNewHdfsDelegationToken(
> DelegationTokenRenewer.java:643)
> at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:
> 488)
> at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer$DelegationTokenRenewerRunnable.
> handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:891)
> at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer$DelegationTokenRenewerRunnable
> .run(DelegationTokenRenewer.java:868)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1742)
> at org.apache.hadoop.crypto.key.kms.KMSClientProvider.
> addDelegationTokens(KMSClientProvider.java:1014)
> ... 16 more
> Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException:
> Authentication failed, URL: http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%
> 40gagan.com&doAs=postgres&user.name=yarn, status: 403, message: Forbidden
> at org.apache.hadoop.security.authentication.client.
> AuthenticatedURL.extractToken(AuthenticatedURL.java:278)
> at org.apache.hadoop.security.authentication.client.
> PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
> at org.apache.hadoop.security.token.delegation.web.
> DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.
> java:132)
> at org.apache.hadoop.security.authentication.client.
> KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212)
> at org.apache.hadoop.security.token.delegation.web.
> DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.
> java:132)
> at org.apache.hadoop.security.authentication.client.
> AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
> at org.apache.hadoop.security.token.delegation.web.
> DelegationTokenAuthenticator.doDelegationTokenOperation(
> DelegationTokenAuthenticator.java:298)
> at org.apache.hadoop.security.token.delegation.web.
> DelegationTokenAuthenticator.getDelegationToken(
> DelegationTokenAuthenticator.java:170)
> at org.apache.hadoop.security.token.delegation.web.
> DelegationTokenAuthenticatedURL.getDelegationToken(
> DelegationTokenAuthenticatedURL.java:371)
> at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(
> KMSClientProvider.java:1019)
> at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(
> KMSClientProvider.java:1014)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1724)
> ... 17 more
>
> ---------------------
>
> The following is what I see in Ranger KMS log (kms.lo)
>
> ---------------------
>
> 2017-03-23 11:02:00,734 DEBUG LimitLatch - Counting
> up[http-bio-9292-Acceptor-0] latch=7
> 2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [uriBC] has
> value [/kms/v1/]
> 2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [semicolon] has
> value [-1]
> 2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [enc] has value
> [ISO-8859-1]
> 2017-03-23 11:02:00,738 DEBUG AuthenticatorBase - Security checking
> request OPTIONS /kms/v1/
> 2017-03-23 11:02:00,738 DEBUG RealmBase - No applicable constraints
> defined
> 2017-03-23 11:02:00,738 DEBUG AuthenticatorBase - Not subject to any
> constraint
> 2017-03-23 11:02:00,738 TRACE StandardWrapper - Returning non-STM
> instance
> 2017-03-23 11:02:00,739 DEBUG Http11Protocol - Socket: [
> org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/
> 192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State
> out: [OPEN]
> 2017-03-23 11:02:00,758 DEBUG Http11Processor - Error parsing HTTP request
> header
> java.io.EOFException: Unexpected EOF read on the socket
> at org.apache.coyote.http11.Http11Processor.
> setRequestLineReadTimeout(Http11Processor.java:169)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(
> AbstractHttp11Processor.java:990)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.
> process(AbstractProtocol.java:625)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.
> run(JIoEndpoint.java:318)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(
> TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
> 2017-03-23 11:02:00,758 DEBUG Http11Protocol - Socket: [
> org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/
> 192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State
> out: [CLOSE
> D]
> 2017-03-23 11:02:00,758 TRACE JIoEndpoint - Closing
> socket:org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/
> 192.168.59.104,port=58547,localport=9292]
>
> ---------------------
>
> The following is from the Ranger KMS access log:
>
> ---------------------
>
> 2017-03-23 11:02:00,738 UNAUTHENTICATED RemoteHost:192.168.59.104
> Method:OPTIONS URL:http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos
> tgres ErrorMsg:'Authentication required'
> 2017-03-23 11:02:00,786 UNAUTHENTICATED RemoteHost:192.168.59.104
> Method:OPTIONS URL:http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos
> tgres&user.name=yarn ErrorMsg:'Authentication required'
>
> ---------------------
>
> The following is from the Ranger KMS audit log (kms-audit.log)
>
> ---------------------
>
> 192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres
> HTTP/1.1" 401 997
> 192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres
> HTTP/1.1" 403 258
> 192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com
> &doAs=postgres&user.name=yarn HTTP/1.1" 401 997
> 192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com
> &doAs=postgres&user.name=yarn HTTP/1.1" 403 258
>
> ---------------------
>
>
> I have added the following proxyuser configuration in Ranger KMS as well:
>
> hadoop.kms.proxyuser.postgres.users=*
> hadoop.kms.proxyuser.postgres.hosts=*
> hadoop.kms.proxyuser.yarn.users=*
> hadoop.kms.proxyuser.yarn.hosts=*
>
> The core-site.xml has the required proxyuser configuration as well:
>
> hadoop.proxyuser.postgres.groups=*
> hadoop.proxyuser.postgres.hosts=*
> hadoop.proxyuser.yarn.groups=*
> hadoop.proxyuser.yarn.hosts=*
>
> But nothing seem to be working in this case here.
>
> I would appreciate some inputs on this one.
>
>
>
> Regards,
> Gagan Brahmi
>