You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@knox.apache.org by "Ruland, Benjamin" <Be...@computacenter.com> on 2016/10/21 08:59:45 UTC
WebHDFS over Knox 0.9.0 not working in secure HDP 2.5 cluster
(Kerberos and SSL)
Hi everyone,
I am experiencing problems with Knox while using WebHDFS in a cluster with Kerberos and SSL.
The KDC is an Microsoft AD 2012. Kerberos-Encryption is set to AES256. Knox is connected to AD via LDAP sync (this is working fine for other Knox services).
I am running HDP 2.5 with Knox 0.9.0
In general, the cluster runs fine. WebHDFS using SPNEGO is working.
But when accessing WebHDFS over Knox, I get an 401 error and some strange logs.
I suspect that Knox is trying to get a ticket for a HTTPS/namenode@REALM principal, which does not exist. Although running SSL, all principals for SPNEGO are HTTP/...
I this a Knox Bug or is this a misconfiguration at some point?
It would be great, if someone has advice.
Best regards,
Benjamin
The used command is:
[root@utilitynode ~]# curl -ik -u validuser "https://utilitynode:8443/gateway/default/webhdfs/v1/?OP=LISTSTATUS"
Enter host password for user 'validuser':
HTTP/1.1 401 Unauthorized
Date: Wed, 12 Oct 2016 07:47:41 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Tue,11-Oct-2016 07:47:41 GMT
WWW-Authenticate: BASIC realm="application"
Content-Length: 0
Server: Jetty(9.2.15.v20160210)
Debug Log in knox gateway.log
2016-10-12 09:51:49,735 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /webhdfs/v1/
2016-10-12 09:51:49,740 DEBUG hadoop.gateway (KnoxLdapRealm.java:getUserDn(673)) - Searching from OU=someOU,DC=somedomain,DC=de where (&(objectclass=person)(sAMAccountName=validuser)) scope subtree
2016-10-12 09:51:49,745 INFO hadoop.gateway (KnoxLdapRealm.java:getUserDn(679)) - Computed userDn: CN=validuser,OU=Users,OU=someOU,DC=somedomain,DC=de using ldapSearch for principal: validuser
2016-10-12 09:51:49,749 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://utilitynode:8443/gateway/default/webhdfs/v1/?OP=LISTSTATUS, direction: IN via explicit rule: WEBHDFS/webhdfs/inbound/namenode/root to URL: https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTSTATUS
2016-10-12 09:51:49,749 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(120)) - Dispatch request: GET https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTSTATUS&doAs=validuser
2016-10-12 09:51:49,781 WARN auth.HttpAuthenticator (HttpAuthenticator.java:generateAuthResponse(207)) - NEGOTIATE authentication error: No valid credentials provided (Mechanism level: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7)))
2016-10-12 09:51:49,782 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(133)) - Dispatch response status: 401
2016-10-12 09:51:49,783 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit character set ISO-8859-1 for entity of type text/html
2016-10-12 09:51:49,783 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound response entity content type: text/html; charset=iso-8859-1
Log in knox gateway.out
Found ticket for knox/utilitynode.somedomain.de@SOMEDOMAIN.DE to go to krbtgt/somedomain.de@SOMEDOMAIN.DE expiring on Wed Oct 12 19:53:51 CEST 2016
Entered Krb5Context.initSecContext with state=STATE_NEW
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
default etypes for default_tgs_enctypes: 18.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
getKDCFromDNS using UDP
>>> KrbKdcReq send: kdc=domaincontroller.somedomain.de. TCP:88, timeout=30000, number of retries =3, #bytes=1661
>>> KDCCommunication: kdc=domaincontroller.somedomain.de. TCP:88, timeout=30000,Attempt =1, #bytes=1661
>>>DEBUG: TCPClient reading 127 bytes
>>> KrbKdcReq send: #bytes read=127
>>> KdcAccessibility: remove domaincontroller.somedomain.de.:88
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
sTime is Wed Oct 12 09:53:51 CEST 2016 1476258831000
suSec is 8354 suSec is 8354
error code is 7
error Message is Server not found in Kerberos database
sname is HTTPS/namenode.somedomain.de@SOMEDOMAIN.DE
msgType is 30
Extracts from topology config:
<topology>
<gateway>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<!-- LDAP Sync properties sit here -->
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
<provider>
<role>authorization</role>
<name>XASecurePDPKnox</name>
<enabled>true</enabled>
</provider>
<provider>
<role>ha</role>
<name>HaProvider</name>
<enabled>true</enabled>
<param>
<name>WEBHDFS</name>
<value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value>
</param>
</provider>
</gateway>
<service>
<role>NAMENODE</role>
<url>hdfs://namenode.somedomain.de:8020</url>
<url>hdfs://namenode2.somedomain.de:8020</url>
</service>
<service>
<role>WEBHDFS</role>
<url>https://namenode.somedomain.de:50470/webhdfs</url>
<url>https://namenode2.somedomain.de:50470/webhdfs</url>
</service>
</topology>
Re: WebHDFS over Knox 0.9.0 not working in secure HDP 2.5 cluster
(Kerberos and SSL)
Posted by larry mccay <la...@gmail.com>.
Glad to hear that it worked!
I will downgrade the dependency to 4.5.1 for the upcoming 0.10.0 release.
Thanks!
On Fri, Oct 21, 2016 at 10:03 AM, Ruland, Benjamin <
Benjamin.Ruland@computacenter.com> wrote:
> Hi Larry,
>
>
>
> I can confirm, that exchanging the httpclient-JAR fixes the problem!
> Thanks for your help!
>
>
>
> I tried with the httpclient-4.4.1.jar (I did not have the 4.5.1 JAR
> around) and the same cURL-Call that failed before now succeeds.
>
>
>
> For reference: I removed the file /usr/hdp/2.5.0.0-1245/knox/dep/httpclient-4.5.2.jar
> and copied the file /usr/hdp/2.5.0.0-1245/falcon/
> client/lib/httpclient-4.4.1.jar to the path /usr/hdp/2.5.0.0-1245/falcon/client/lib/
> instead. After restarting Knox, I tested again.
>
>
>
> Thanks again,
>
> Benjamin
>
>
>
>
>
> *Von:* larry mccay [mailto:lmccay@apache.org]
> *Gesendet:* Freitag, 21. Oktober 2016 15:44
> *An:* user@knox.apache.org
> *Betreff:* Re: WebHDFS over Knox 0.9.0 not working in secure HDP 2.5
> cluster (Kerberos and SSL)
>
>
>
> You can try to workaround this by replacing the dep/httpclient-4.5.2.jar
> jar with the 4.5.1 version.
>
> 4.5.1 did not include the change that broke 4.5.2.
>
> Hopefully, there is no use of any incompatible changes in our dispatch
> classes.
>
>
>
> Please let me know if you can get by this error with this workaround.
>
>
>
> On Fri, Oct 21, 2016 at 8:38 AM, larry mccay <lm...@apache.org> wrote:
>
> From what I can see, this problem is directly related to:
> https://issues.apache.org/jira/browse/HTTPCLIENT-1712.
>
>
>
> I have asked them to provide a release that removes this incorrect patch
> but we will likely have to deal with it in Knox - if at all possible.
>
> I will look into overriding GGSSchemeBase in Knox and figure out how to
> use the extension or forked version as a downloadable patch.
>
>
>
> Sorry for the inconvenience!
>
>
>
> On Fri, Oct 21, 2016 at 7:48 AM, larry mccay <lm...@apache.org> wrote:
>
> Hi Benjamin -
>
>
>
> I suspect, based on the error message, that you are right.
>
> The HTTP service name in the SPN is incorrectly set as HTTPS.
>
>
>
> Not sure why this would be.
>
> I will look into our kerberos dispatch code and see if we are explicitly
> setting this for some reason.
>
> We should be just letting HttpClient do it for us but I will check.
>
>
>
> thanks,
>
>
>
> --larry
>
>
>
> On Fri, Oct 21, 2016 at 4:59 AM, Ruland, Benjamin <Benjamin.Ruland@
> computacenter.com> wrote:
>
> Hi everyone,
>
>
>
> I am experiencing problems with Knox while using WebHDFS in a cluster with
> Kerberos and SSL.
>
> The KDC is an Microsoft AD 2012. Kerberos-Encryption is set to AES256.
> Knox is connected to AD via LDAP sync (this is working fine for other Knox
> services).
>
> I am running HDP 2.5 with Knox 0.9.0
>
>
>
> In general, the cluster runs fine. WebHDFS using SPNEGO is working.
>
>
>
> But when accessing WebHDFS over Knox, I get an 401 error and some strange
> logs.
>
> I suspect that Knox is trying to get a ticket for a HTTPS/namenode@REALM
> principal, which does not exist. Although running SSL, all principals for
> SPNEGO are HTTP/...
>
>
>
> I this a Knox Bug or is this a misconfiguration at some point?
>
>
>
> It would be great, if someone has advice.
>
>
>
> Best regards,
>
> Benjamin
>
>
>
>
>
>
>
>
>
>
>
> The used command is:
>
>
>
> [root@utilitynode ~]# curl -ik -u validuser "https://utilitynode:8443/
> gateway/default/webhdfs/v1/?OP=LISTSTATUS"
>
> Enter host password for user 'validuser':
>
> HTTP/1.1 401 Unauthorized
>
> Date: Wed, 12 Oct 2016 07:47:41 GMT
>
> Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0;
> Expires=Tue,11-Oct-2016 07:47:41 GMT
>
> WWW-Authenticate: BASIC realm="application"
>
> Content-Length: 0
>
> Server: Jetty(9.2.15.v20160210)
>
>
>
>
>
> Debug Log in knox gateway.log
>
>
>
> 2016-10-12 09:51:49,735 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116))
> - Received request: GET /webhdfs/v1/
>
> 2016-10-12 09:51:49,740 DEBUG hadoop.gateway (KnoxLdapRealm.java:getUserDn(673))
> - Searching from OU=someOU,DC=somedomain,DC=de where (&(objectclass=person)(sAMAccountName=validuser))
> scope subtree
>
> 2016-10-12 09:51:49,745 INFO hadoop.gateway (KnoxLdapRealm.java:getUserDn(679))
> - Computed userDn: CN=validuser,OU=Users,OU=someOU,DC=somedomain,DC=de
> using ldapSearch for principal: validuser
>
> 2016-10-12 09:51:49,749 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(166))
> - Rewrote URL: https://utilitynode:8443/gateway/default/webhdfs/v1/?
> OP=LISTSTATUS, direction: IN via explicit rule: WEBHDFS/webhdfs/inbound/namenode/root
> to URL: https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTSTATUS
>
> 2016-10-12 09:51:49,749 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(120))
> - Dispatch request: GET https://utilitynode.somedomain.de:50470/webhdfs/
> v1/?OP=LISTSTATUS&doAs=validuser
>
> 2016-10-12 09:51:49,781 WARN auth.HttpAuthenticator
> (HttpAuthenticator.java:generateAuthResponse(207)) - NEGOTIATE
> authentication error: No valid credentials provided (Mechanism level: No
> valid credentials provided (Mechanism level: Server not found in Kerberos
> database (7)))
>
> 2016-10-12 09:51:49,782 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(133))
> - Dispatch response status: 401
>
> 2016-10-12 09:51:49,783 DEBUG hadoop.gateway (DefaultDispatch.java:
> getInboundResponseContentType(202)) - Using explicit character set
> ISO-8859-1 for entity of type text/html
>
> 2016-10-12 09:51:49,783 DEBUG hadoop.gateway (DefaultDispatch.java:
> getInboundResponseContentType(210)) - Inbound response entity content
> type: text/html; charset=iso-8859-1
>
>
>
>
>
> Log in knox gateway.out
>
>
>
> Found ticket for knox/utilitynode.somedomain.de@SOMEDOMAIN.DE to go to
> krbtgt/somedomain.de@SOMEDOMAIN.DE expiring on Wed Oct 12 19:53:51 CEST
> 2016
>
> Entered Krb5Context.initSecContext with state=STATE_NEW
>
> Service ticket not found in the subject
>
> >>> Credentials acquireServiceCreds: same realm
>
> default etypes for default_tgs_enctypes: 18.
>
> >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>
> >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>
> getKDCFromDNS using UDP
>
> >>> KrbKdcReq send: kdc=domaincontroller.somedomain.de. TCP:88,
> timeout=30000, number of retries =3, #bytes=1661
>
> >>> KDCCommunication: kdc=domaincontroller.somedomain.de. TCP:88,
> timeout=30000,Attempt =1, #bytes=1661
>
> >>>DEBUG: TCPClient reading 127 bytes
>
> >>> KrbKdcReq send: #bytes read=127
>
> >>> KdcAccessibility: remove domaincontroller.somedomain.de.:88
>
> >>> KDCRep: init() encoding tag is 126 req type is 13
>
> >>>KRBError:
>
> sTime is Wed Oct 12 09:53:51 CEST 2016 1476258831000
>
> suSec is 8354 suSec is 8354
>
> error code is 7
>
> error Message is Server not found in Kerberos database
>
> sname is HTTPS/namenode.somedomain.de@SOMEDOMAIN.DE
>
> msgType is 30
>
>
>
>
>
> Extracts from topology config:
>
>
>
> <topology>
>
>
>
> <gateway>
>
>
>
> <provider>
>
> <role>authentication</role>
>
> <name>ShiroProvider</name>
>
> <enabled>true</enabled>
>
>
>
> <!-- LDAP Sync properties sit here -->
>
>
>
> <provider>
>
> <role>identity-assertion</role>
>
> <name>Default</name>
>
> <enabled>true</enabled>
>
> </provider>
>
>
>
> <provider>
>
> <role>authorization</role>
>
> <name>XASecurePDPKnox</name>
>
> <enabled>true</enabled>
>
> </provider>
>
>
>
> <provider>
>
> <role>ha</role>
>
> <name>HaProvider</name>
>
> <enabled>true</enabled>
>
> <param>
>
> <name>WEBHDFS</name>
>
> <value>maxFailoverAttempts=3;failoverSleep=1000;
> maxRetryAttempts=300;retrySleep=1000;enabled=true</value>
>
> </param>
>
> </provider>
>
>
>
> </gateway>
>
>
>
> <service>
>
> <role>NAMENODE</role>
>
> <url>hdfs://namenode.somedomain.de:8020</url>
>
> <url>hdfs://namenode2.somedomain.de:8020</url>
>
> </service>
>
>
>
> <service>
>
> <role>WEBHDFS</role>
>
> <url>https://namenode.somedomain.de:50470/webhdfs</url>
>
> <url>https://namenode2.somedomain.de:50470/webhdfs</url>
>
> </service>
>
>
>
> </topology>
>
>
>
>
> -----------------------------------
> Computacenter AG & Co. oHG, mit Sitz in Kerpen
> (Amtsgericht Köln HRA 18096)
> Vertretungsberechtigte Gesellschafter:
> Computacenter Aktiengesellschaft, mit Sitz in Köln (Amtsgericht Köln HRB 28384)
> Vorstand: Tony Conophy
> Aufsichtsrat: Michael Norris (Vorsitzender)
> Computacenter Management GmbH, mit Sitz in Köln (Amtsgericht Köln HRB 28284)
> Geschäftsführer: Dr. Karsten Freihube, Dr. Thomas Kottmann, Reiner Louis, Thomas Jescheck
> Visit us on the Internet: http://www.computacenter.de
> Visit our Online-Shop: https://shop.computacenter.de
>
> This email is confidential. If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this mail in error, please tell us immediately by return email and delete the document.
> -----------------------------------
>
>
>
>
>
>
>
>
> -----------------------------------
> Computacenter AG & Co. oHG, mit Sitz in Kerpen
> (Amtsgericht Köln HRA 18096)
> Vertretungsberechtigte Gesellschafter:
> Computacenter Aktiengesellschaft, mit Sitz in Köln (Amtsgericht Köln HRB 28384)
> Vorstand: Tony Conophy
> Aufsichtsrat: Michael Norris (Vorsitzender)
> Computacenter Management GmbH, mit Sitz in Köln (Amtsgericht Köln HRB 28284)
> Geschäftsführer: Dr. Karsten Freihube, Dr. Thomas Kottmann, Reiner Louis, Thomas Jescheck
> Visit us on the Internet: http://www.computacenter.de
> Visit our Online-Shop: https://shop.computacenter.de
>
> This email is confidential. If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this mail in error, please tell us immediately by return email and delete the document.
> -----------------------------------
>
>
AW: WebHDFS over Knox 0.9.0 not working in secure HDP 2.5 cluster
(Kerberos and SSL)
Posted by "Ruland, Benjamin" <Be...@computacenter.com>.
Hi Larry,
I can confirm, that exchanging the httpclient-JAR fixes the problem! Thanks for your help!
I tried with the httpclient-4.4.1.jar (I did not have the 4.5.1 JAR around) and the same cURL-Call that failed before now succeeds.
For reference: I removed the file /usr/hdp/2.5.0.0-1245/knox/dep/httpclient-4.5.2.jar and copied the file /usr/hdp/2.5.0.0-1245/falcon/client/lib/httpclient-4.4.1.jar to the path /usr/hdp/2.5.0.0-1245/falcon/client/lib/ instead. After restarting Knox, I tested again.
Thanks again,
Benjamin
Von: larry mccay [mailto:lmccay@apache.org]
Gesendet: Freitag, 21. Oktober 2016 15:44
An: user@knox.apache.org
Betreff: Re: WebHDFS over Knox 0.9.0 not working in secure HDP 2.5 cluster (Kerberos and SSL)
You can try to workaround this by replacing the dep/httpclient-4.5.2.jar jar with the 4.5.1 version.
4.5.1 did not include the change that broke 4.5.2.
Hopefully, there is no use of any incompatible changes in our dispatch classes.
Please let me know if you can get by this error with this workaround.
On Fri, Oct 21, 2016 at 8:38 AM, larry mccay <lm...@apache.org>> wrote:
From what I can see, this problem is directly related to: https://issues.apache.org/jira/browse/HTTPCLIENT-1712.
I have asked them to provide a release that removes this incorrect patch but we will likely have to deal with it in Knox - if at all possible.
I will look into overriding GGSSchemeBase in Knox and figure out how to use the extension or forked version as a downloadable patch.
Sorry for the inconvenience!
On Fri, Oct 21, 2016 at 7:48 AM, larry mccay <lm...@apache.org>> wrote:
Hi Benjamin -
I suspect, based on the error message, that you are right.
The HTTP service name in the SPN is incorrectly set as HTTPS.
Not sure why this would be.
I will look into our kerberos dispatch code and see if we are explicitly setting this for some reason.
We should be just letting HttpClient do it for us but I will check.
thanks,
--larry
On Fri, Oct 21, 2016 at 4:59 AM, Ruland, Benjamin <Be...@computacenter.com>> wrote:
Hi everyone,
I am experiencing problems with Knox while using WebHDFS in a cluster with Kerberos and SSL.
The KDC is an Microsoft AD 2012. Kerberos-Encryption is set to AES256. Knox is connected to AD via LDAP sync (this is working fine for other Knox services).
I am running HDP 2.5 with Knox 0.9.0
In general, the cluster runs fine. WebHDFS using SPNEGO is working.
But when accessing WebHDFS over Knox, I get an 401 error and some strange logs.
I suspect that Knox is trying to get a ticket for a HTTPS/namenode@REALM principal, which does not exist. Although running SSL, all principals for SPNEGO are HTTP/...
I this a Knox Bug or is this a misconfiguration at some point?
It would be great, if someone has advice.
Best regards,
Benjamin
The used command is:
[root@utilitynode ~]# curl -ik -u validuser "https://utilitynode:8443/gateway/default/webhdfs/v1/?OP=LISTSTATUS"
Enter host password for user 'validuser':
HTTP/1.1 401 Unauthorized
Date: Wed, 12 Oct 2016 07:47:41 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Tue,11-Oct-2016 07:47:41 GMT
WWW-Authenticate: BASIC realm="application"
Content-Length: 0
Server: Jetty(9.2.15.v20160210)
Debug Log in knox gateway.log
2016-10-12 09:51:49,735 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /webhdfs/v1/
2016-10-12 09:51:49,740 DEBUG hadoop.gateway (KnoxLdapRealm.java:getUserDn(673)) - Searching from OU=someOU,DC=somedomain,DC=de where (&(objectclass=person)(sAMAccountName=validuser)) scope subtree
2016-10-12 09:51:49,745 INFO hadoop.gateway (KnoxLdapRealm.java:getUserDn(679)) - Computed userDn: CN=validuser,OU=Users,OU=someOU,DC=somedomain,DC=de using ldapSearch for principal: validuser
2016-10-12 09:51:49,749 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://utilitynode:8443/gateway/default/webhdfs/v1/?OP=LISTSTATUS, direction: IN via explicit rule: WEBHDFS/webhdfs/inbound/namenode/root to URL: https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTSTATUS
2016-10-12 09:51:49,749 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(120)) - Dispatch request: GET https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTSTATUS&doAs=validuser
2016-10-12 09:51:49,781 WARN auth.HttpAuthenticator (HttpAuthenticator.java:generateAuthResponse(207)) - NEGOTIATE authentication error: No valid credentials provided (Mechanism level: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7)))
2016-10-12 09:51:49,782 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(133)) - Dispatch response status: 401
2016-10-12 09:51:49,783 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit character set ISO-8859-1 for entity of type text/html
2016-10-12 09:51:49,783 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound response entity content type: text/html; charset=iso-8859-1
Log in knox gateway.out
Found ticket for knox/utilitynode.somedomain.de@SOMEDOMAIN.DE<ma...@SOMEDOMAIN.DE> to go to krbtgt/somedomain.de@SOMEDOMAIN.DE<ma...@SOMEDOMAIN.DE> expiring on Wed Oct 12 19:53:51 CEST 2016
Entered Krb5Context.initSecContext with state=STATE_NEW
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
default etypes for default_tgs_enctypes: 18.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
getKDCFromDNS using UDP
>>> KrbKdcReq send: kdc=domaincontroller.somedomain.de<http://domaincontroller.somedomain.de>. TCP:88, timeout=30000, number of retries =3, #bytes=1661
>>> KDCCommunication: kdc=domaincontroller.somedomain.de<http://domaincontroller.somedomain.de>. TCP:88, timeout=30000,Attempt =1, #bytes=1661
>>>DEBUG: TCPClient reading 127 bytes
>>> KrbKdcReq send: #bytes read=127
>>> KdcAccessibility: remove domaincontroller.somedomain.de<http://domaincontroller.somedomain.de>.:88
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
sTime is Wed Oct 12 09:53:51 CEST 2016 1476258831000
suSec is 8354 suSec is 8354
error code is 7
error Message is Server not found in Kerberos database
sname is HTTPS/namenode.somedomain.de@SOMEDOMAIN.DE<ma...@SOMEDOMAIN.DE>
msgType is 30
Extracts from topology config:
<topology>
<gateway>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<!-- LDAP Sync properties sit here -->
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
<provider>
<role>authorization</role>
<name>XASecurePDPKnox</name>
<enabled>true</enabled>
</provider>
<provider>
<role>ha</role>
<name>HaProvider</name>
<enabled>true</enabled>
<param>
<name>WEBHDFS</name>
<value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value>
</param>
</provider>
</gateway>
<service>
<role>NAMENODE</role>
<url>hdfs://namenode.somedomain.de:8020<http://namenode.somedomain.de:8020></url>
<url>hdfs://namenode2.somedomain.de:8020<http://namenode2.somedomain.de:8020></url>
</service>
<service>
<role>WEBHDFS</role>
<url>https://namenode.somedomain.de:50470/webhdfs</url>
<url>https://namenode2.somedomain.de:50470/webhdfs</url>
</service>
</topology>
-----------------------------------
Computacenter AG & Co. oHG, mit Sitz in Kerpen
(Amtsgericht Köln HRA 18096)
Vertretungsberechtigte Gesellschafter:
Computacenter Aktiengesellschaft, mit Sitz in Köln (Amtsgericht Köln HRB 28384)
Vorstand: Tony Conophy
Aufsichtsrat: Michael Norris (Vorsitzender)
Computacenter Management GmbH, mit Sitz in Köln (Amtsgericht Köln HRB 28284)
Geschäftsführer: Dr. Karsten Freihube, Dr. Thomas Kottmann, Reiner Louis, Thomas Jescheck
Visit us on the Internet: http://www.computacenter.de<http://www.computacenter.de/>
Visit our Online-Shop: https://shop.computacenter.de<https://shop.computacenter.de/>
This email is confidential. If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this mail in error, please tell us immediately by return email and delete the document.
-----------------------------------
Re: WebHDFS over Knox 0.9.0 not working in secure HDP 2.5 cluster
(Kerberos and SSL)
Posted by larry mccay <lm...@apache.org>.
You can try to workaround this by replacing the dep/httpclient-4.5.2.jar
jar with the 4.5.1 version.
4.5.1 did not include the change that broke 4.5.2.
Hopefully, there is no use of any incompatible changes in our dispatch
classes.
Please let me know if you can get by this error with this workaround.
On Fri, Oct 21, 2016 at 8:38 AM, larry mccay <lm...@apache.org> wrote:
> From what I can see, this problem is directly related to:
> https://issues.apache.org/jira/browse/HTTPCLIENT-1712.
>
> I have asked them to provide a release that removes this incorrect patch
> but we will likely have to deal with it in Knox - if at all possible.
> I will look into overriding GGSSchemeBase in Knox and figure out how to
> use the extension or forked version as a downloadable patch.
>
> Sorry for the inconvenience!
>
> On Fri, Oct 21, 2016 at 7:48 AM, larry mccay <lm...@apache.org> wrote:
>
>> Hi Benjamin -
>>
>> I suspect, based on the error message, that you are right.
>> The HTTP service name in the SPN is incorrectly set as HTTPS.
>>
>> Not sure why this would be.
>> I will look into our kerberos dispatch code and see if we are explicitly
>> setting this for some reason.
>> We should be just letting HttpClient do it for us but I will check.
>>
>> thanks,
>>
>> --larry
>>
>> On Fri, Oct 21, 2016 at 4:59 AM, Ruland, Benjamin <
>> Benjamin.Ruland@computacenter.com> wrote:
>>
>>> Hi everyone,
>>>
>>>
>>>
>>> I am experiencing problems with Knox while using WebHDFS in a cluster
>>> with Kerberos and SSL.
>>>
>>> The KDC is an Microsoft AD 2012. Kerberos-Encryption is set to AES256.
>>> Knox is connected to AD via LDAP sync (this is working fine for other Knox
>>> services).
>>>
>>> I am running HDP 2.5 with Knox 0.9.0
>>>
>>>
>>>
>>> In general, the cluster runs fine. WebHDFS using SPNEGO is working.
>>>
>>>
>>>
>>> But when accessing WebHDFS over Knox, I get an 401 error and some
>>> strange logs.
>>>
>>> I suspect that Knox is trying to get a ticket for a HTTPS/namenode@REALM
>>> principal, which does not exist. Although running SSL, all principals for
>>> SPNEGO are HTTP/...
>>>
>>>
>>>
>>> I this a Knox Bug or is this a misconfiguration at some point?
>>>
>>>
>>>
>>> It would be great, if someone has advice.
>>>
>>>
>>>
>>> Best regards,
>>>
>>> Benjamin
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> The used command is:
>>>
>>>
>>>
>>> [root@utilitynode ~]# curl -ik -u validuser "
>>> https://utilitynode:8443/gateway/default/webhdfs/v1/?OP=LISTSTATUS"
>>>
>>> Enter host password for user 'validuser':
>>>
>>> HTTP/1.1 401 Unauthorized
>>>
>>> Date: Wed, 12 Oct 2016 07:47:41 GMT
>>>
>>> Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0;
>>> Expires=Tue,11-Oct-2016 07:47:41 GMT
>>>
>>> WWW-Authenticate: BASIC realm="application"
>>>
>>> Content-Length: 0
>>>
>>> Server: Jetty(9.2.15.v20160210)
>>>
>>>
>>>
>>>
>>>
>>> Debug Log in knox gateway.log
>>>
>>>
>>>
>>> 2016-10-12 09:51:49,735 DEBUG hadoop.gateway
>>> (GatewayFilter.java:doFilter(116)) - Received request: GET /webhdfs/v1/
>>>
>>> 2016-10-12 09:51:49,740 DEBUG hadoop.gateway
>>> (KnoxLdapRealm.java:getUserDn(673)) - Searching from
>>> OU=someOU,DC=somedomain,DC=de where (&(objectclass=person)(sAMAccountName=validuser))
>>> scope subtree
>>>
>>> 2016-10-12 09:51:49,745 INFO hadoop.gateway
>>> (KnoxLdapRealm.java:getUserDn(679)) - Computed userDn:
>>> CN=validuser,OU=Users,OU=someOU,DC=somedomain,DC=de using ldapSearch
>>> for principal: validuser
>>>
>>> 2016-10-12 09:51:49,749 DEBUG hadoop.gateway
>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL:
>>> https://utilitynode:8443/gateway/default/webhdfs/v1/?OP=LISTSTATUS,
>>> direction: IN via explicit rule: WEBHDFS/webhdfs/inbound/namenode/root
>>> to URL: https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTS
>>> TATUS
>>>
>>> 2016-10-12 09:51:49,749 DEBUG hadoop.gateway
>>> (DefaultDispatch.java:executeOutboundRequest(120)) - Dispatch request:
>>> GET https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTS
>>> TATUS&doAs=validuser
>>>
>>> 2016-10-12 09:51:49,781 WARN auth.HttpAuthenticator
>>> (HttpAuthenticator.java:generateAuthResponse(207)) - NEGOTIATE
>>> authentication error: No valid credentials provided (Mechanism level: No
>>> valid credentials provided (Mechanism level: Server not found in Kerberos
>>> database (7)))
>>>
>>> 2016-10-12 09:51:49,782 DEBUG hadoop.gateway
>>> (DefaultDispatch.java:executeOutboundRequest(133)) - Dispatch response
>>> status: 401
>>>
>>> 2016-10-12 09:51:49,783 DEBUG hadoop.gateway
>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using
>>> explicit character set ISO-8859-1 for entity of type text/html
>>>
>>> 2016-10-12 09:51:49,783 DEBUG hadoop.gateway
>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>> response entity content type: text/html; charset=iso-8859-1
>>>
>>>
>>>
>>>
>>>
>>> Log in knox gateway.out
>>>
>>>
>>>
>>> Found ticket for knox/utilitynode.somedomain.de@SOMEDOMAIN.DE to go to
>>> krbtgt/somedomain.de@SOMEDOMAIN.DE expiring on Wed Oct 12 19:53:51 CEST
>>> 2016
>>>
>>> Entered Krb5Context.initSecContext with state=STATE_NEW
>>>
>>> Service ticket not found in the subject
>>>
>>> >>> Credentials acquireServiceCreds: same realm
>>>
>>> default etypes for default_tgs_enctypes: 18.
>>>
>>> >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>>
>>> >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>>
>>> getKDCFromDNS using UDP
>>>
>>> >>> KrbKdcReq send: kdc=domaincontroller.somedomain.de. TCP:88,
>>> timeout=30000, number of retries =3, #bytes=1661
>>>
>>> >>> KDCCommunication: kdc=domaincontroller.somedomain.de. TCP:88,
>>> timeout=30000,Attempt =1, #bytes=1661
>>>
>>> >>>DEBUG: TCPClient reading 127 bytes
>>>
>>> >>> KrbKdcReq send: #bytes read=127
>>>
>>> >>> KdcAccessibility: remove domaincontroller.somedomain.de.:88
>>>
>>> >>> KDCRep: init() encoding tag is 126 req type is 13
>>>
>>> >>>KRBError:
>>>
>>> sTime is Wed Oct 12 09:53:51 CEST 2016 1476258831000
>>>
>>> suSec is 8354 suSec is 8354
>>>
>>> error code is 7
>>>
>>> error Message is Server not found in Kerberos database
>>>
>>> sname is HTTPS/namenode.somedomain.de@SOMEDOMAIN.DE
>>>
>>> msgType is 30
>>>
>>>
>>>
>>>
>>>
>>> Extracts from topology config:
>>>
>>>
>>>
>>> <topology>
>>>
>>>
>>>
>>> <gateway>
>>>
>>>
>>>
>>> <provider>
>>>
>>> <role>authentication</role>
>>>
>>> <name>ShiroProvider</name>
>>>
>>> <enabled>true</enabled>
>>>
>>>
>>>
>>> <!-- LDAP Sync properties sit here -->
>>>
>>>
>>>
>>> <provider>
>>>
>>> <role>identity-assertion</role>
>>>
>>> <name>Default</name>
>>>
>>> <enabled>true</enabled>
>>>
>>> </provider>
>>>
>>>
>>>
>>> <provider>
>>>
>>> <role>authorization</role>
>>>
>>> <name>XASecurePDPKnox</name>
>>>
>>> <enabled>true</enabled>
>>>
>>> </provider>
>>>
>>>
>>>
>>> <provider>
>>>
>>> <role>ha</role>
>>>
>>> <name>HaProvider</name>
>>>
>>> <enabled>true</enabled>
>>>
>>> <param>
>>>
>>> <name>WEBHDFS</name>
>>>
>>> <value>maxFailoverAttempts=3;failoverSleep=1000;maxRe
>>> tryAttempts=300;retrySleep=1000;enabled=true</value>
>>>
>>> </param>
>>>
>>> </provider>
>>>
>>>
>>>
>>> </gateway>
>>>
>>>
>>>
>>> <service>
>>>
>>> <role>NAMENODE</role>
>>>
>>> <url>hdfs://namenode.somedomain.de:8020</url>
>>>
>>> <url>hdfs://namenode2.somedomain.de:8020</url>
>>>
>>> </service>
>>>
>>>
>>>
>>> <service>
>>>
>>> <role>WEBHDFS</role>
>>>
>>> <url>https://namenode.somedomain.de:50470/webhdfs</url>
>>>
>>> <url>https://namenode2.somedomain.de:50470/webhdfs</url>
>>>
>>> </service>
>>>
>>>
>>>
>>> </topology>
>>>
>>>
>>> -----------------------------------
>>> Computacenter AG & Co. oHG, mit Sitz in Kerpen
>>> (Amtsgericht Köln HRA 18096)
>>> Vertretungsberechtigte Gesellschafter:
>>> Computacenter Aktiengesellschaft, mit Sitz in Köln (Amtsgericht Köln HRB 28384)
>>> Vorstand: Tony Conophy
>>> Aufsichtsrat: Michael Norris (Vorsitzender)
>>> Computacenter Management GmbH, mit Sitz in Köln (Amtsgericht Köln HRB 28284)
>>> Geschäftsführer: Dr. Karsten Freihube, Dr. Thomas Kottmann, Reiner Louis, Thomas Jescheck
>>> Visit us on the Internet: http://www.computacenter.de
>>> Visit our Online-Shop: https://shop.computacenter.de
>>>
>>> This email is confidential. If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this mail in error, please tell us immediately by return email and delete the document.
>>> -----------------------------------
>>>
>>>
>>
>
Re: WebHDFS over Knox 0.9.0 not working in secure HDP 2.5 cluster
(Kerberos and SSL)
Posted by larry mccay <lm...@apache.org>.
From what I can see, this problem is directly related to:
https://issues.apache.org/jira/browse/HTTPCLIENT-1712.
I have asked them to provide a release that removes this incorrect patch
but we will likely have to deal with it in Knox - if at all possible.
I will look into overriding GGSSchemeBase in Knox and figure out how to use
the extension or forked version as a downloadable patch.
Sorry for the inconvenience!
On Fri, Oct 21, 2016 at 7:48 AM, larry mccay <lm...@apache.org> wrote:
> Hi Benjamin -
>
> I suspect, based on the error message, that you are right.
> The HTTP service name in the SPN is incorrectly set as HTTPS.
>
> Not sure why this would be.
> I will look into our kerberos dispatch code and see if we are explicitly
> setting this for some reason.
> We should be just letting HttpClient do it for us but I will check.
>
> thanks,
>
> --larry
>
> On Fri, Oct 21, 2016 at 4:59 AM, Ruland, Benjamin <Benjamin.Ruland@
> computacenter.com> wrote:
>
>> Hi everyone,
>>
>>
>>
>> I am experiencing problems with Knox while using WebHDFS in a cluster
>> with Kerberos and SSL.
>>
>> The KDC is an Microsoft AD 2012. Kerberos-Encryption is set to AES256.
>> Knox is connected to AD via LDAP sync (this is working fine for other Knox
>> services).
>>
>> I am running HDP 2.5 with Knox 0.9.0
>>
>>
>>
>> In general, the cluster runs fine. WebHDFS using SPNEGO is working.
>>
>>
>>
>> But when accessing WebHDFS over Knox, I get an 401 error and some strange
>> logs.
>>
>> I suspect that Knox is trying to get a ticket for a HTTPS/namenode@REALM
>> principal, which does not exist. Although running SSL, all principals for
>> SPNEGO are HTTP/...
>>
>>
>>
>> I this a Knox Bug or is this a misconfiguration at some point?
>>
>>
>>
>> It would be great, if someone has advice.
>>
>>
>>
>> Best regards,
>>
>> Benjamin
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> The used command is:
>>
>>
>>
>> [root@utilitynode ~]# curl -ik -u validuser "
>> https://utilitynode:8443/gateway/default/webhdfs/v1/?OP=LISTSTATUS"
>>
>> Enter host password for user 'validuser':
>>
>> HTTP/1.1 401 Unauthorized
>>
>> Date: Wed, 12 Oct 2016 07:47:41 GMT
>>
>> Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0;
>> Expires=Tue,11-Oct-2016 07:47:41 GMT
>>
>> WWW-Authenticate: BASIC realm="application"
>>
>> Content-Length: 0
>>
>> Server: Jetty(9.2.15.v20160210)
>>
>>
>>
>>
>>
>> Debug Log in knox gateway.log
>>
>>
>>
>> 2016-10-12 09:51:49,735 DEBUG hadoop.gateway
>> (GatewayFilter.java:doFilter(116)) - Received request: GET /webhdfs/v1/
>>
>> 2016-10-12 09:51:49,740 DEBUG hadoop.gateway
>> (KnoxLdapRealm.java:getUserDn(673)) - Searching from
>> OU=someOU,DC=somedomain,DC=de where (&(objectclass=person)(sAMAccountName=validuser))
>> scope subtree
>>
>> 2016-10-12 09:51:49,745 INFO hadoop.gateway
>> (KnoxLdapRealm.java:getUserDn(679)) - Computed userDn:
>> CN=validuser,OU=Users,OU=someOU,DC=somedomain,DC=de using ldapSearch for
>> principal: validuser
>>
>> 2016-10-12 09:51:49,749 DEBUG hadoop.gateway
>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL:
>> https://utilitynode:8443/gateway/default/webhdfs/v1/?OP=LISTSTATUS,
>> direction: IN via explicit rule: WEBHDFS/webhdfs/inbound/namenode/root
>> to URL: https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTSTATUS
>>
>> 2016-10-12 09:51:49,749 DEBUG hadoop.gateway
>> (DefaultDispatch.java:executeOutboundRequest(120)) - Dispatch request:
>> GET https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=
>> LISTSTATUS&doAs=validuser
>>
>> 2016-10-12 09:51:49,781 WARN auth.HttpAuthenticator
>> (HttpAuthenticator.java:generateAuthResponse(207)) - NEGOTIATE
>> authentication error: No valid credentials provided (Mechanism level: No
>> valid credentials provided (Mechanism level: Server not found in Kerberos
>> database (7)))
>>
>> 2016-10-12 09:51:49,782 DEBUG hadoop.gateway
>> (DefaultDispatch.java:executeOutboundRequest(133)) - Dispatch response
>> status: 401
>>
>> 2016-10-12 09:51:49,783 DEBUG hadoop.gateway
>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using
>> explicit character set ISO-8859-1 for entity of type text/html
>>
>> 2016-10-12 09:51:49,783 DEBUG hadoop.gateway
>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>> response entity content type: text/html; charset=iso-8859-1
>>
>>
>>
>>
>>
>> Log in knox gateway.out
>>
>>
>>
>> Found ticket for knox/utilitynode.somedomain.de@SOMEDOMAIN.DE to go to
>> krbtgt/somedomain.de@SOMEDOMAIN.DE expiring on Wed Oct 12 19:53:51 CEST
>> 2016
>>
>> Entered Krb5Context.initSecContext with state=STATE_NEW
>>
>> Service ticket not found in the subject
>>
>> >>> Credentials acquireServiceCreds: same realm
>>
>> default etypes for default_tgs_enctypes: 18.
>>
>> >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>
>> >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>
>> getKDCFromDNS using UDP
>>
>> >>> KrbKdcReq send: kdc=domaincontroller.somedomain.de. TCP:88,
>> timeout=30000, number of retries =3, #bytes=1661
>>
>> >>> KDCCommunication: kdc=domaincontroller.somedomain.de. TCP:88,
>> timeout=30000,Attempt =1, #bytes=1661
>>
>> >>>DEBUG: TCPClient reading 127 bytes
>>
>> >>> KrbKdcReq send: #bytes read=127
>>
>> >>> KdcAccessibility: remove domaincontroller.somedomain.de.:88
>>
>> >>> KDCRep: init() encoding tag is 126 req type is 13
>>
>> >>>KRBError:
>>
>> sTime is Wed Oct 12 09:53:51 CEST 2016 1476258831000
>>
>> suSec is 8354 suSec is 8354
>>
>> error code is 7
>>
>> error Message is Server not found in Kerberos database
>>
>> sname is HTTPS/namenode.somedomain.de@SOMEDOMAIN.DE
>>
>> msgType is 30
>>
>>
>>
>>
>>
>> Extracts from topology config:
>>
>>
>>
>> <topology>
>>
>>
>>
>> <gateway>
>>
>>
>>
>> <provider>
>>
>> <role>authentication</role>
>>
>> <name>ShiroProvider</name>
>>
>> <enabled>true</enabled>
>>
>>
>>
>> <!-- LDAP Sync properties sit here -->
>>
>>
>>
>> <provider>
>>
>> <role>identity-assertion</role>
>>
>> <name>Default</name>
>>
>> <enabled>true</enabled>
>>
>> </provider>
>>
>>
>>
>> <provider>
>>
>> <role>authorization</role>
>>
>> <name>XASecurePDPKnox</name>
>>
>> <enabled>true</enabled>
>>
>> </provider>
>>
>>
>>
>> <provider>
>>
>> <role>ha</role>
>>
>> <name>HaProvider</name>
>>
>> <enabled>true</enabled>
>>
>> <param>
>>
>> <name>WEBHDFS</name>
>>
>> <value>maxFailoverAttempts=3;failoverSleep=1000;maxRe
>> tryAttempts=300;retrySleep=1000;enabled=true</value>
>>
>> </param>
>>
>> </provider>
>>
>>
>>
>> </gateway>
>>
>>
>>
>> <service>
>>
>> <role>NAMENODE</role>
>>
>> <url>hdfs://namenode.somedomain.de:8020</url>
>>
>> <url>hdfs://namenode2.somedomain.de:8020</url>
>>
>> </service>
>>
>>
>>
>> <service>
>>
>> <role>WEBHDFS</role>
>>
>> <url>https://namenode.somedomain.de:50470/webhdfs</url>
>>
>> <url>https://namenode2.somedomain.de:50470/webhdfs</url>
>>
>> </service>
>>
>>
>>
>> </topology>
>>
>>
>> -----------------------------------
>> Computacenter AG & Co. oHG, mit Sitz in Kerpen
>> (Amtsgericht Köln HRA 18096)
>> Vertretungsberechtigte Gesellschafter:
>> Computacenter Aktiengesellschaft, mit Sitz in Köln (Amtsgericht Köln HRB 28384)
>> Vorstand: Tony Conophy
>> Aufsichtsrat: Michael Norris (Vorsitzender)
>> Computacenter Management GmbH, mit Sitz in Köln (Amtsgericht Köln HRB 28284)
>> Geschäftsführer: Dr. Karsten Freihube, Dr. Thomas Kottmann, Reiner Louis, Thomas Jescheck
>> Visit us on the Internet: http://www.computacenter.de
>> Visit our Online-Shop: https://shop.computacenter.de
>>
>> This email is confidential. If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this mail in error, please tell us immediately by return email and delete the document.
>> -----------------------------------
>>
>>
>
Re: WebHDFS over Knox 0.9.0 not working in secure HDP 2.5 cluster
(Kerberos and SSL)
Posted by larry mccay <lm...@apache.org>.
Hi Benjamin -
I suspect, based on the error message, that you are right.
The HTTP service name in the SPN is incorrectly set as HTTPS.
Not sure why this would be.
I will look into our kerberos dispatch code and see if we are explicitly
setting this for some reason.
We should be just letting HttpClient do it for us but I will check.
thanks,
--larry
On Fri, Oct 21, 2016 at 4:59 AM, Ruland, Benjamin <
Benjamin.Ruland@computacenter.com> wrote:
> Hi everyone,
>
>
>
> I am experiencing problems with Knox while using WebHDFS in a cluster with
> Kerberos and SSL.
>
> The KDC is an Microsoft AD 2012. Kerberos-Encryption is set to AES256.
> Knox is connected to AD via LDAP sync (this is working fine for other Knox
> services).
>
> I am running HDP 2.5 with Knox 0.9.0
>
>
>
> In general, the cluster runs fine. WebHDFS using SPNEGO is working.
>
>
>
> But when accessing WebHDFS over Knox, I get an 401 error and some strange
> logs.
>
> I suspect that Knox is trying to get a ticket for a HTTPS/namenode@REALM
> principal, which does not exist. Although running SSL, all principals for
> SPNEGO are HTTP/...
>
>
>
> I this a Knox Bug or is this a misconfiguration at some point?
>
>
>
> It would be great, if someone has advice.
>
>
>
> Best regards,
>
> Benjamin
>
>
>
>
>
>
>
>
>
>
>
> The used command is:
>
>
>
> [root@utilitynode ~]# curl -ik -u validuser "https://utilitynode:8443/
> gateway/default/webhdfs/v1/?OP=LISTSTATUS"
>
> Enter host password for user 'validuser':
>
> HTTP/1.1 401 Unauthorized
>
> Date: Wed, 12 Oct 2016 07:47:41 GMT
>
> Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0;
> Expires=Tue,11-Oct-2016 07:47:41 GMT
>
> WWW-Authenticate: BASIC realm="application"
>
> Content-Length: 0
>
> Server: Jetty(9.2.15.v20160210)
>
>
>
>
>
> Debug Log in knox gateway.log
>
>
>
> 2016-10-12 09:51:49,735 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116))
> - Received request: GET /webhdfs/v1/
>
> 2016-10-12 09:51:49,740 DEBUG hadoop.gateway (KnoxLdapRealm.java:getUserDn(673))
> - Searching from OU=someOU,DC=somedomain,DC=de where (&(objectclass=person)(sAMAccountName=validuser))
> scope subtree
>
> 2016-10-12 09:51:49,745 INFO hadoop.gateway (KnoxLdapRealm.java:getUserDn(679))
> - Computed userDn: CN=validuser,OU=Users,OU=someOU,DC=somedomain,DC=de
> using ldapSearch for principal: validuser
>
> 2016-10-12 09:51:49,749 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(166))
> - Rewrote URL: https://utilitynode:8443/gateway/default/webhdfs/v1/?
> OP=LISTSTATUS, direction: IN via explicit rule: WEBHDFS/webhdfs/inbound/namenode/root
> to URL: https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTSTATUS
>
> 2016-10-12 09:51:49,749 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(120))
> - Dispatch request: GET https://utilitynode.somedomain.de:50470/webhdfs/
> v1/?OP=LISTSTATUS&doAs=validuser
>
> 2016-10-12 09:51:49,781 WARN auth.HttpAuthenticator
> (HttpAuthenticator.java:generateAuthResponse(207)) - NEGOTIATE
> authentication error: No valid credentials provided (Mechanism level: No
> valid credentials provided (Mechanism level: Server not found in Kerberos
> database (7)))
>
> 2016-10-12 09:51:49,782 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(133))
> - Dispatch response status: 401
>
> 2016-10-12 09:51:49,783 DEBUG hadoop.gateway (DefaultDispatch.java:
> getInboundResponseContentType(202)) - Using explicit character set
> ISO-8859-1 for entity of type text/html
>
> 2016-10-12 09:51:49,783 DEBUG hadoop.gateway (DefaultDispatch.java:
> getInboundResponseContentType(210)) - Inbound response entity content
> type: text/html; charset=iso-8859-1
>
>
>
>
>
> Log in knox gateway.out
>
>
>
> Found ticket for knox/utilitynode.somedomain.de@SOMEDOMAIN.DE to go to
> krbtgt/somedomain.de@SOMEDOMAIN.DE expiring on Wed Oct 12 19:53:51 CEST
> 2016
>
> Entered Krb5Context.initSecContext with state=STATE_NEW
>
> Service ticket not found in the subject
>
> >>> Credentials acquireServiceCreds: same realm
>
> default etypes for default_tgs_enctypes: 18.
>
> >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>
> >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>
> getKDCFromDNS using UDP
>
> >>> KrbKdcReq send: kdc=domaincontroller.somedomain.de. TCP:88,
> timeout=30000, number of retries =3, #bytes=1661
>
> >>> KDCCommunication: kdc=domaincontroller.somedomain.de. TCP:88,
> timeout=30000,Attempt =1, #bytes=1661
>
> >>>DEBUG: TCPClient reading 127 bytes
>
> >>> KrbKdcReq send: #bytes read=127
>
> >>> KdcAccessibility: remove domaincontroller.somedomain.de.:88
>
> >>> KDCRep: init() encoding tag is 126 req type is 13
>
> >>>KRBError:
>
> sTime is Wed Oct 12 09:53:51 CEST 2016 1476258831000
>
> suSec is 8354 suSec is 8354
>
> error code is 7
>
> error Message is Server not found in Kerberos database
>
> sname is HTTPS/namenode.somedomain.de@SOMEDOMAIN.DE
>
> msgType is 30
>
>
>
>
>
> Extracts from topology config:
>
>
>
> <topology>
>
>
>
> <gateway>
>
>
>
> <provider>
>
> <role>authentication</role>
>
> <name>ShiroProvider</name>
>
> <enabled>true</enabled>
>
>
>
> <!-- LDAP Sync properties sit here -->
>
>
>
> <provider>
>
> <role>identity-assertion</role>
>
> <name>Default</name>
>
> <enabled>true</enabled>
>
> </provider>
>
>
>
> <provider>
>
> <role>authorization</role>
>
> <name>XASecurePDPKnox</name>
>
> <enabled>true</enabled>
>
> </provider>
>
>
>
> <provider>
>
> <role>ha</role>
>
> <name>HaProvider</name>
>
> <enabled>true</enabled>
>
> <param>
>
> <name>WEBHDFS</name>
>
> <value>maxFailoverAttempts=3;failoverSleep=1000;
> maxRetryAttempts=300;retrySleep=1000;enabled=true</value>
>
> </param>
>
> </provider>
>
>
>
> </gateway>
>
>
>
> <service>
>
> <role>NAMENODE</role>
>
> <url>hdfs://namenode.somedomain.de:8020</url>
>
> <url>hdfs://namenode2.somedomain.de:8020</url>
>
> </service>
>
>
>
> <service>
>
> <role>WEBHDFS</role>
>
> <url>https://namenode.somedomain.de:50470/webhdfs</url>
>
> <url>https://namenode2.somedomain.de:50470/webhdfs</url>
>
> </service>
>
>
>
> </topology>
>
>
> -----------------------------------
> Computacenter AG & Co. oHG, mit Sitz in Kerpen
> (Amtsgericht Köln HRA 18096)
> Vertretungsberechtigte Gesellschafter:
> Computacenter Aktiengesellschaft, mit Sitz in Köln (Amtsgericht Köln HRB 28384)
> Vorstand: Tony Conophy
> Aufsichtsrat: Michael Norris (Vorsitzender)
> Computacenter Management GmbH, mit Sitz in Köln (Amtsgericht Köln HRB 28284)
> Geschäftsführer: Dr. Karsten Freihube, Dr. Thomas Kottmann, Reiner Louis, Thomas Jescheck
> Visit us on the Internet: http://www.computacenter.de
> Visit our Online-Shop: https://shop.computacenter.de
>
> This email is confidential. If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this mail in error, please tell us immediately by return email and delete the document.
> -----------------------------------
>
>