You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ranger.apache.org by Aaron Gresch <ag...@gmail.com> on 2017/09/19 18:58:09 UTC
HDFS Kerberos documentation/setup
1) What documentation should I be following to install Ranger manually for
a Kerberos Hadoop cluster? I am interested in the HDFS Plugin.
This is what I found linked from the apache site, but is very old:
https://cwiki.apache.org/confluence/display/RANGER/Ranger+Installation+Guide
2) Following those instructions, I see "Create a repository in Ranger
Policy Manager. E.g. "local_hdfs"."
Is this the same as creating a Service? I see Services under HDFS on the
Ranger admin server.
3) Creating an HDFS service lists a Username and Password. We don't use
passwords for our clusters, but have keytabs. What should this mandatory
field be? What is it used for?
4) How is this supposed to be setup for secure clusters? Is there any
manually setup example I can be pointed to?
Thanks,
Aaron
Re: HDFS Kerberos documentation/setup
Posted by Ramesh Mani <rm...@hortonworks.com>.
Aaron,
Also check on xa_portal.log is there any exceptio, if the login to fetch the policies for the hdfs plugin fails it might have returned a html form error.
Regards,
Ramesh
From: Ramesh Mani <rm...@hortonworks.com>>
Date: Friday, September 22, 2017 at 8:38 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
Yes, the looks like son deserialization issue. Please check the libraries in /hadoop/lib/ and ranger ews/webapp/WEB-INF/lib folder jersey-json*jar is there and all same version.
Regards,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Friday, September 22, 2017 at 6:34 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
Apparently the Admin server is sending back an HTML form to have the name node logon. I'm not sure how this is supposed to work. I assume the HDFS plugin would take care of this somehow before syncing the policies or throw an exception that it could not log on?
Can someone explain what I should be doing to make this work?
Thanks,
Aaron
On Thu, Sep 21, 2017 at 3:50 PM, Aaron Gresch <ag...@gmail.com>> wrote:
I encountered an error "Class org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer not found" when starting the name node. I added the patch for RANGER-1412 to my enable-pdfs-plugin.sh script and got past this error.
Now when I start the namenode, I see this callstack:
https://github.com/apache/ranger/blob/ranger-0.7/agents-common/src/main/java/org/apache/ranger/admin/client/RangerAdminRESTClient.java#L141
2017-09-21 20:43:33,571 [main] ERROR util.PolicyRefresher: PolicyRefresher(serviceName=openqe79blue): failed to refresh policies. Will continue to use last known version of policies (-1)
com.sun.jersey.api.client.ClientHandlerException: A message body reader for Java class org.apache.ranger.plugin.util.ServicePolicies, and Java type class org.apache.ranger.plugin.util.ServicePolicies, and MIME media type text/html; charset=ISO-8859-1 was not found
at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:549)
at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:506)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:141)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:149)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:157)
at org.apache.ranger.authorization.hadoop.RangerHdfsPlugin.init(RangerHdfsAuthorizer.java:613)
at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:98)
at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:86)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1131)
at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:760)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:711)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:905)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:884)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1678)
On Wed, Sep 20, 2017 at 4:59 PM, Ramesh Mani <rm...@hortonworks.com>> wrote:
Aron,
Also make sure that the ranger admin conf files /etc/ranger/admin/conf/ranger-admin-site.xml has these rangerlookup kerberos principal. Should be there after installation.
Regards,
Ramesh
From: Ramesh Mani <rm...@hortonworks.com>>
Date: Wednesday, September 20, 2017 at 1:32 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
Aron,
TestConnection is just used for lookup purpose only. (To list the resource while maintaining policies). There should be steps to create keytab for rangerlookup, just make sure that you have policy for that user so it can list the hdfs directories/files.
Even if the test connection fails it doesn’t stop you from maintaining policies and using ranger.
Regards,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, September 20, 2017 at 1:10 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
When I am in the Ranger Admin Service Manager -> Edit Service for HDFS, there is a Test Connection button. When I press it, it tries to login with Username and Password. We use keytabs. Tracing the ranger_admin.log, in BaseClient.java, the lookupPrincipal and lookupKeytab are not set. If I force these to be set in the code, it then uses a keytab.
I'm not certain how the keytabs are to be specified other than the install.properties file. Clearly I must not have specified them properly.
https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80
<https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80>
On Wed, Sep 20, 2017 at 3:00 PM, Ramesh Mani <rm...@hortonworks.com>> wrote:
Aron,
When you say Login in with user and password is that Ranger Admin UI login? Or is the hdfs plugin login into ranger to fetch the policy?
Looks like the NPE is not related to Ranger, but please check namenode.log what is there. Please enable debug on namenode and check it out.
Thanks,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, September 20, 2017 at 12:21 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
Thanks.
Having lots of issues trying to get this to work.
Issue 1 - Admin Server
I'm not exactly sure what I am doing right or wrong so far, but it is still trying to login with a user and password rather than a keytab. In BaseClient.login(), I hard-coded the keytab and principal, and then I see a proper HDFS file listing occurring. This however is failing (see issue 2). Looks like it is expecting some xalogin.xml that does not exist to set these properties. I still need to dig into why this does not exist.
I'm not clear what authentication mode means exactly. I don't think UNIX/LDAP/AD fit anything we do based on questioning the Hadoop team here. I'm not exactly clear what this setting is used for or which setting we should specify.
Issue 2 - Namenode
I installed the plugin and was able to restart the name node, but no policy data was in the cache directory, it appears unable to sync. When I do a "hadoop ls" on the command line, I get a NPE:
Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:238)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:243)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:182)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:499)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1605)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1623)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePath(FSDirectory.java:544)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getListingInt(FSDirStatAndListingOp.java:55)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:3695)
On Tue, Sep 19, 2017 at 2:53 PM, Ramesh Mani <rm...@hortonworks.com>> wrote:
Hi Aaron
Please check this out https://cwiki.apache.org/confluence/display/RANGER/Ranger+installation+in+Kerberized++Environment
Regards,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Tuesday, September 19, 2017 at 11:58 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: HDFS Kerberos documentation/setup
1) What documentation should I be following to install Ranger manually for a Kerberos Hadoop cluster? I am interested in the HDFS Plugin.
This is what I found linked from the apache site, but is very old:
https://cwiki.apache.org/confluence/display/RANGER/Ranger+Installation+Guide
2) Following those instructions, I see "Create a repository in Ranger Policy Manager. E.g. "local_hdfs"."
Is this the same as creating a Service? I see Services under HDFS on the Ranger admin server.
3) Creating an HDFS service lists a Username and Password. We don't use passwords for our clusters, but have keytabs. What should this mandatory field be? What is it used for?
4) How is this supposed to be setup for secure clusters? Is there any manually setup example I can be pointed to?
Thanks,
Aaron
Re: HDFS Kerberos documentation/setup
Posted by Ramesh Mani <rm...@hortonworks.com>.
Yes, the looks like son deserialization issue. Please check the libraries in /hadoop/lib/ and ranger ews/webapp/WEB-INF/lib folder jersey-json*jar is there and all same version.
Regards,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Friday, September 22, 2017 at 6:34 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
Apparently the Admin server is sending back an HTML form to have the name node logon. I'm not sure how this is supposed to work. I assume the HDFS plugin would take care of this somehow before syncing the policies or throw an exception that it could not log on?
Can someone explain what I should be doing to make this work?
Thanks,
Aaron
On Thu, Sep 21, 2017 at 3:50 PM, Aaron Gresch <ag...@gmail.com>> wrote:
I encountered an error "Class org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer not found" when starting the name node. I added the patch for RANGER-1412 to my enable-pdfs-plugin.sh script and got past this error.
Now when I start the namenode, I see this callstack:
https://github.com/apache/ranger/blob/ranger-0.7/agents-common/src/main/java/org/apache/ranger/admin/client/RangerAdminRESTClient.java#L141
2017-09-21 20:43:33,571 [main] ERROR util.PolicyRefresher: PolicyRefresher(serviceName=openqe79blue): failed to refresh policies. Will continue to use last known version of policies (-1)
com.sun.jersey.api.client.ClientHandlerException: A message body reader for Java class org.apache.ranger.plugin.util.ServicePolicies, and Java type class org.apache.ranger.plugin.util.ServicePolicies, and MIME media type text/html; charset=ISO-8859-1 was not found
at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:549)
at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:506)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:141)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:149)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:157)
at org.apache.ranger.authorization.hadoop.RangerHdfsPlugin.init(RangerHdfsAuthorizer.java:613)
at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:98)
at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:86)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1131)
at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:760)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:711)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:905)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:884)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1678)
On Wed, Sep 20, 2017 at 4:59 PM, Ramesh Mani <rm...@hortonworks.com>> wrote:
Aron,
Also make sure that the ranger admin conf files /etc/ranger/admin/conf/ranger-admin-site.xml has these rangerlookup kerberos principal. Should be there after installation.
Regards,
Ramesh
From: Ramesh Mani <rm...@hortonworks.com>>
Date: Wednesday, September 20, 2017 at 1:32 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
Aron,
TestConnection is just used for lookup purpose only. (To list the resource while maintaining policies). There should be steps to create keytab for rangerlookup, just make sure that you have policy for that user so it can list the hdfs directories/files.
Even if the test connection fails it doesn’t stop you from maintaining policies and using ranger.
Regards,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, September 20, 2017 at 1:10 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
When I am in the Ranger Admin Service Manager -> Edit Service for HDFS, there is a Test Connection button. When I press it, it tries to login with Username and Password. We use keytabs. Tracing the ranger_admin.log, in BaseClient.java, the lookupPrincipal and lookupKeytab are not set. If I force these to be set in the code, it then uses a keytab.
I'm not certain how the keytabs are to be specified other than the install.properties file. Clearly I must not have specified them properly.
https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80
<https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80>
On Wed, Sep 20, 2017 at 3:00 PM, Ramesh Mani <rm...@hortonworks.com>> wrote:
Aron,
When you say Login in with user and password is that Ranger Admin UI login? Or is the hdfs plugin login into ranger to fetch the policy?
Looks like the NPE is not related to Ranger, but please check namenode.log what is there. Please enable debug on namenode and check it out.
Thanks,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, September 20, 2017 at 12:21 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
Thanks.
Having lots of issues trying to get this to work.
Issue 1 - Admin Server
I'm not exactly sure what I am doing right or wrong so far, but it is still trying to login with a user and password rather than a keytab. In BaseClient.login(), I hard-coded the keytab and principal, and then I see a proper HDFS file listing occurring. This however is failing (see issue 2). Looks like it is expecting some xalogin.xml that does not exist to set these properties. I still need to dig into why this does not exist.
I'm not clear what authentication mode means exactly. I don't think UNIX/LDAP/AD fit anything we do based on questioning the Hadoop team here. I'm not exactly clear what this setting is used for or which setting we should specify.
Issue 2 - Namenode
I installed the plugin and was able to restart the name node, but no policy data was in the cache directory, it appears unable to sync. When I do a "hadoop ls" on the command line, I get a NPE:
Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:238)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:243)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:182)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:499)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1605)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1623)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePath(FSDirectory.java:544)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getListingInt(FSDirStatAndListingOp.java:55)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:3695)
On Tue, Sep 19, 2017 at 2:53 PM, Ramesh Mani <rm...@hortonworks.com>> wrote:
Hi Aaron
Please check this out https://cwiki.apache.org/confluence/display/RANGER/Ranger+installation+in+Kerberized++Environment
Regards,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Tuesday, September 19, 2017 at 11:58 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: HDFS Kerberos documentation/setup
1) What documentation should I be following to install Ranger manually for a Kerberos Hadoop cluster? I am interested in the HDFS Plugin.
This is what I found linked from the apache site, but is very old:
https://cwiki.apache.org/confluence/display/RANGER/Ranger+Installation+Guide
2) Following those instructions, I see "Create a repository in Ranger Policy Manager. E.g. "local_hdfs"."
Is this the same as creating a Service? I see Services under HDFS on the Ranger admin server.
3) Creating an HDFS service lists a Username and Password. We don't use passwords for our clusters, but have keytabs. What should this mandatory field be? What is it used for?
4) How is this supposed to be setup for secure clusters? Is there any manually setup example I can be pointed to?
Thanks,
Aaron
Re: HDFS Kerberos documentation/setup
Posted by Aaron Gresch <ag...@gmail.com>.
Apparently the Admin server is sending back an HTML form to have the name
node logon. I'm not sure how this is supposed to work. I assume the HDFS
plugin would take care of this somehow before syncing the policies or throw
an exception that it could not log on?
Can someone explain what I should be doing to make this work?
Thanks,
Aaron
On Thu, Sep 21, 2017 at 3:50 PM, Aaron Gresch <ag...@gmail.com> wrote:
>
> I encountered an error "Class org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer
> not found" when starting the name node. I added the patch for RANGER-1412
> to my enable-pdfs-plugin.sh script and got past this error.
>
> Now when I start the namenode, I see this callstack:
>
> https://github.com/apache/ranger/blob/ranger-0.7/agents-
> common/src/main/java/org/apache/ranger/admin/client/
> RangerAdminRESTClient.java#L141
>
>
> 2017-09-21 20:43:33,571 [main] ERROR util.PolicyRefresher:
> PolicyRefresher(serviceName=openqe79blue): failed to refresh policies.
> Will continue to use last known version of policies (-1)
>
> com.sun.jersey.api.client.ClientHandlerException: A message body reader
> for Java class org.apache.ranger.plugin.util.ServicePolicies, and Java
> type class org.apache.ranger.plugin.util.ServicePolicies, and MIME media
> type text/html; charset=ISO-8859-1 was not found
>
> at com.sun.jersey.api.client.ClientResponse.getEntity(
> ClientResponse.java:549)
>
> at com.sun.jersey.api.client.ClientResponse.getEntity(
> ClientResponse.java:506)
>
> at org.apache.ranger.admin.client.RangerAdminRESTClient.
> getServicePoliciesIfUpdated(RangerAdminRESTClient.java:141)
>
> at org.apache.ranger.plugin.util.PolicyRefresher.
> loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
>
> at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(
> PolicyRefresher.java:202)
>
> at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(
> PolicyRefresher.java:149)
>
> at org.apache.ranger.plugin.service.RangerBasePlugin.init(
> RangerBasePlugin.java:157)
>
> at org.apache.ranger.authorization.hadoop.RangerHdfsPlugin.init(
> RangerHdfsAuthorizer.java:613)
>
> at org.apache.ranger.authorization.hadoop.
> RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:98)
>
> at org.apache.ranger.authorization.hadoop.
> RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:86)
>
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> startCommonServices(FSNamesystem.java:1131)
>
> at org.apache.hadoop.hdfs.server.namenode.NameNode.
> startCommonServices(NameNode.java:760)
>
> at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(
> NameNode.java:711)
>
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(
> NameNode.java:905)
>
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(
> NameNode.java:884)
>
> at org.apache.hadoop.hdfs.server.namenode.NameNode.
> createNameNode(NameNode.java:1610)
>
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(
> NameNode.java:1678)
>
>
> On Wed, Sep 20, 2017 at 4:59 PM, Ramesh Mani <rm...@hortonworks.com>
> wrote:
>
>> Aron,
>>
>> Also make sure that the ranger admin conf files /etc/ranger/admin/conf/ranger-admin-site.xml
>> has these rangerlookup kerberos principal. Should be there after
>> installation.
>>
>> Regards,
>> Ramesh
>>
>> From: Ramesh Mani <rm...@hortonworks.com>
>> Date: Wednesday, September 20, 2017 at 1:32 PM
>>
>> To: "user@ranger.apache.org" <us...@ranger.apache.org>
>> Subject: Re: HDFS Kerberos documentation/setup
>>
>> Aron,
>>
>> TestConnection is just used for lookup purpose only. (To list the
>> resource while maintaining policies). There should be steps to create
>> keytab for rangerlookup, just make sure that you have policy for that
>> user so it can list the hdfs directories/files.
>>
>> Even if the test connection fails it doesn’t stop you from maintaining
>> policies and using ranger.
>>
>> Regards,
>> Ramesh
>>
>> From: Aaron Gresch <ag...@gmail.com>
>> Reply-To: "user@ranger.apache.org" <us...@ranger.apache.org>
>> Date: Wednesday, September 20, 2017 at 1:10 PM
>> To: "user@ranger.apache.org" <us...@ranger.apache.org>
>> Subject: Re: HDFS Kerberos documentation/setup
>>
>>
>> When I am in the Ranger Admin Service Manager -> Edit Service for HDFS,
>> there is a Test Connection button. When I press it, it tries to login with
>> Username and Password. We use keytabs. Tracing the ranger_admin.log, in
>> BaseClient.java, the lookupPrincipal and lookupKeytab are not set. If I
>> force these to be set in the code, it then uses a keytab.
>>
>> I'm not certain how the keytabs are to be specified other than the
>> install.properties file. Clearly I must not have specified them properly.
>>
>> https://github.com/apache/ranger/blob/688807cf74fc434e246a2f
>> 7d6c0e71e941178421/agents-common/src/main/java/org/
>> apache/ranger/plugin/client/BaseClient.java#L79-L80
>>
>>
>> <https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80>
>>
>>
>> On Wed, Sep 20, 2017 at 3:00 PM, Ramesh Mani <rm...@hortonworks.com>
>> wrote:
>>
>>> Aron,
>>>
>>> When you say Login in with user and password is that Ranger Admin UI
>>> login? Or is the hdfs plugin login into ranger to fetch the policy?
>>>
>>> Looks like the NPE is not related to Ranger, but please check
>>> namenode.log what is there. Please enable debug on namenode and check it
>>> out.
>>>
>>> Thanks,
>>> Ramesh
>>>
>>> From: Aaron Gresch <ag...@gmail.com>
>>> Reply-To: "user@ranger.apache.org" <us...@ranger.apache.org>
>>> Date: Wednesday, September 20, 2017 at 12:21 PM
>>> To: "user@ranger.apache.org" <us...@ranger.apache.org>
>>> Subject: Re: HDFS Kerberos documentation/setup
>>>
>>>
>>> Thanks.
>>>
>>> Having lots of issues trying to get this to work.
>>>
>>> Issue 1 - Admin Server
>>>
>>> I'm not exactly sure what I am doing right or wrong so far, but it is
>>> still trying to login with a user and password rather than a keytab. In
>>> BaseClient.login(), I hard-coded the keytab and principal, and then I see a
>>> proper HDFS file listing occurring. This however is failing (see issue
>>> 2). Looks like it is expecting some xalogin.xml that does not exist to set
>>> these properties. I still need to dig into why this does not exist.
>>>
>>> I'm not clear what authentication mode means exactly. I don't think
>>> UNIX/LDAP/AD fit anything we do based on questioning the Hadoop team here.
>>> I'm not exactly clear what this setting is used for or which setting we
>>> should specify.
>>>
>>> Issue 2 - Namenode
>>>
>>> I installed the plugin and was able to restart the name node, but no
>>> policy data was in the cache directory, it appears unable to sync. When I
>>> do a "hadoop ls" on the command line, I get a NPE:
>>>
>>> Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
>>> java.lang.NullPointerException
>>>
>>> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:238)
>>>
>>> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.g
>>> etINodeAttrs(FSPermissionChecker.java:243)
>>>
>>> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.c
>>> heckPermission(FSPermissionChecker.java:182)
>>>
>>> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.c
>>> heckTraverse(FSPermissionChecker.java:499)
>>>
>>> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTrav
>>> erse(FSDirectory.java:1605)
>>>
>>> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTrav
>>> erse(FSDirectory.java:1623)
>>>
>>> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePa
>>> th(FSDirectory.java:544)
>>>
>>> at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp
>>> .getListingInt(FSDirStatAndListingOp.java:55)
>>>
>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListi
>>> ng(FSNamesystem.java:3695)
>>>
>>>
>>> On Tue, Sep 19, 2017 at 2:53 PM, Ramesh Mani <rm...@hortonworks.com>
>>> wrote:
>>>
>>>> Hi Aaron
>>>>
>>>> Please check this out https://cwiki.apache.org/c
>>>> onfluence/display/RANGER/Ranger+installation+in+Kerberized++Environment
>>>>
>>>> Regards,
>>>> Ramesh
>>>>
>>>> From: Aaron Gresch <ag...@gmail.com>
>>>> Reply-To: "user@ranger.apache.org" <us...@ranger.apache.org>
>>>> Date: Tuesday, September 19, 2017 at 11:58 AM
>>>> To: "user@ranger.apache.org" <us...@ranger.apache.org>
>>>> Subject: HDFS Kerberos documentation/setup
>>>>
>>>> 1) What documentation should I be following to install Ranger manually
>>>> for a Kerberos Hadoop cluster? I am interested in the HDFS Plugin.
>>>>
>>>> This is what I found linked from the apache site, but is very old:
>>>>
>>>> https://cwiki.apache.org/confluence/display/RANGER/Ranger+In
>>>> stallation+Guide
>>>>
>>>>
>>>> 2) Following those instructions, I see "Create a repository in Ranger
>>>> Policy Manager. E.g. "local_hdfs"."
>>>>
>>>> Is this the same as creating a Service? I see Services under HDFS on
>>>> the Ranger admin server.
>>>>
>>>>
>>>> 3) Creating an HDFS service lists a Username and Password. We don't
>>>> use passwords for our clusters, but have keytabs. What should this
>>>> mandatory field be? What is it used for?
>>>>
>>>>
>>>> 4) How is this supposed to be setup for secure clusters? Is there any
>>>> manually setup example I can be pointed to?
>>>>
>>>>
>>>> Thanks,
>>>> Aaron
>>>>
>>>
>>>
>>
>
Re: HDFS Kerberos documentation/setup
Posted by Aaron Gresch <ag...@gmail.com>.
I encountered an error "Class
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer not found" when
starting the name node. I added the patch for RANGER-1412 to my
enable-pdfs-plugin.sh script and got past this error.
Now when I start the namenode, I see this callstack:
https://github.com/apache/ranger/blob/ranger-0.7/agents-common/src/main/java/org/apache/ranger/admin/client/RangerAdminRESTClient.java#L141
2017-09-21 20:43:33,571 [main] ERROR util.PolicyRefresher:
PolicyRefresher(serviceName=openqe79blue): failed to refresh policies. Will
continue to use last known version of policies (-1)
com.sun.jersey.api.client.ClientHandlerException: A message body reader for
Java class org.apache.ranger.plugin.util.ServicePolicies, and Java type
class org.apache.ranger.plugin.util.ServicePolicies, and MIME media type
text/html; charset=ISO-8859-1 was not found
at
com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:549)
at
com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:506)
at
org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:141)
at
org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at
org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at
org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:149)
at
org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:157)
at
org.apache.ranger.authorization.hadoop.RangerHdfsPlugin.init(RangerHdfsAuthorizer.java:613)
at
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:98)
at
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:86)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1131)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:760)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:711)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:905)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:884)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1678)
On Wed, Sep 20, 2017 at 4:59 PM, Ramesh Mani <rm...@hortonworks.com> wrote:
> Aron,
>
> Also make sure that the ranger admin conf files /etc/ranger/admin/conf/ranger-admin-site.xml
> has these rangerlookup kerberos principal. Should be there after
> installation.
>
> Regards,
> Ramesh
>
> From: Ramesh Mani <rm...@hortonworks.com>
> Date: Wednesday, September 20, 2017 at 1:32 PM
>
> To: "user@ranger.apache.org" <us...@ranger.apache.org>
> Subject: Re: HDFS Kerberos documentation/setup
>
> Aron,
>
> TestConnection is just used for lookup purpose only. (To list the resource
> while maintaining policies). There should be steps to create keytab for rangerlookup,
> just make sure that you have policy for that user so it can list the hdfs
> directories/files.
>
> Even if the test connection fails it doesn’t stop you from maintaining
> policies and using ranger.
>
> Regards,
> Ramesh
>
> From: Aaron Gresch <ag...@gmail.com>
> Reply-To: "user@ranger.apache.org" <us...@ranger.apache.org>
> Date: Wednesday, September 20, 2017 at 1:10 PM
> To: "user@ranger.apache.org" <us...@ranger.apache.org>
> Subject: Re: HDFS Kerberos documentation/setup
>
>
> When I am in the Ranger Admin Service Manager -> Edit Service for HDFS,
> there is a Test Connection button. When I press it, it tries to login with
> Username and Password. We use keytabs. Tracing the ranger_admin.log, in
> BaseClient.java, the lookupPrincipal and lookupKeytab are not set. If I
> force these to be set in the code, it then uses a keytab.
>
> I'm not certain how the keytabs are to be specified other than the
> install.properties file. Clearly I must not have specified them properly.
>
> https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71
> e941178421/agents-common/src/main/java/org/apache/ranger/
> plugin/client/BaseClient.java#L79-L80
>
>
> <https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80>
>
>
> On Wed, Sep 20, 2017 at 3:00 PM, Ramesh Mani <rm...@hortonworks.com>
> wrote:
>
>> Aron,
>>
>> When you say Login in with user and password is that Ranger Admin UI
>> login? Or is the hdfs plugin login into ranger to fetch the policy?
>>
>> Looks like the NPE is not related to Ranger, but please check
>> namenode.log what is there. Please enable debug on namenode and check it
>> out.
>>
>> Thanks,
>> Ramesh
>>
>> From: Aaron Gresch <ag...@gmail.com>
>> Reply-To: "user@ranger.apache.org" <us...@ranger.apache.org>
>> Date: Wednesday, September 20, 2017 at 12:21 PM
>> To: "user@ranger.apache.org" <us...@ranger.apache.org>
>> Subject: Re: HDFS Kerberos documentation/setup
>>
>>
>> Thanks.
>>
>> Having lots of issues trying to get this to work.
>>
>> Issue 1 - Admin Server
>>
>> I'm not exactly sure what I am doing right or wrong so far, but it is
>> still trying to login with a user and password rather than a keytab. In
>> BaseClient.login(), I hard-coded the keytab and principal, and then I see a
>> proper HDFS file listing occurring. This however is failing (see issue
>> 2). Looks like it is expecting some xalogin.xml that does not exist to set
>> these properties. I still need to dig into why this does not exist.
>>
>> I'm not clear what authentication mode means exactly. I don't think
>> UNIX/LDAP/AD fit anything we do based on questioning the Hadoop team here.
>> I'm not exactly clear what this setting is used for or which setting we
>> should specify.
>>
>> Issue 2 - Namenode
>>
>> I installed the plugin and was able to restart the name node, but no
>> policy data was in the cache directory, it appears unable to sync. When I
>> do a "hadoop ls" on the command line, I get a NPE:
>>
>> Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
>> java.lang.NullPointerException
>>
>> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:238)
>>
>> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.g
>> etINodeAttrs(FSPermissionChecker.java:243)
>>
>> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.c
>> heckPermission(FSPermissionChecker.java:182)
>>
>> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.c
>> heckTraverse(FSPermissionChecker.java:499)
>>
>> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTrav
>> erse(FSDirectory.java:1605)
>>
>> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTrav
>> erse(FSDirectory.java:1623)
>>
>> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePa
>> th(FSDirectory.java:544)
>>
>> at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp
>> .getListingInt(FSDirStatAndListingOp.java:55)
>>
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListi
>> ng(FSNamesystem.java:3695)
>>
>>
>> On Tue, Sep 19, 2017 at 2:53 PM, Ramesh Mani <rm...@hortonworks.com>
>> wrote:
>>
>>> Hi Aaron
>>>
>>> Please check this out https://cwiki.apache.org/c
>>> onfluence/display/RANGER/Ranger+installation+in+Kerberized++Environment
>>>
>>> Regards,
>>> Ramesh
>>>
>>> From: Aaron Gresch <ag...@gmail.com>
>>> Reply-To: "user@ranger.apache.org" <us...@ranger.apache.org>
>>> Date: Tuesday, September 19, 2017 at 11:58 AM
>>> To: "user@ranger.apache.org" <us...@ranger.apache.org>
>>> Subject: HDFS Kerberos documentation/setup
>>>
>>> 1) What documentation should I be following to install Ranger manually
>>> for a Kerberos Hadoop cluster? I am interested in the HDFS Plugin.
>>>
>>> This is what I found linked from the apache site, but is very old:
>>>
>>> https://cwiki.apache.org/confluence/display/RANGER/Ranger+In
>>> stallation+Guide
>>>
>>>
>>> 2) Following those instructions, I see "Create a repository in Ranger
>>> Policy Manager. E.g. "local_hdfs"."
>>>
>>> Is this the same as creating a Service? I see Services under HDFS on
>>> the Ranger admin server.
>>>
>>>
>>> 3) Creating an HDFS service lists a Username and Password. We don't use
>>> passwords for our clusters, but have keytabs. What should this mandatory
>>> field be? What is it used for?
>>>
>>>
>>> 4) How is this supposed to be setup for secure clusters? Is there any
>>> manually setup example I can be pointed to?
>>>
>>>
>>> Thanks,
>>> Aaron
>>>
>>
>>
>
Re: HDFS Kerberos documentation/setup
Posted by Ramesh Mani <rm...@hortonworks.com>.
Aron,
Also make sure that the ranger admin conf files /etc/ranger/admin/conf/ranger-admin-site.xml has these rangerlookup kerberos principal. Should be there after installation.
Regards,
Ramesh
From: Ramesh Mani <rm...@hortonworks.com>>
Date: Wednesday, September 20, 2017 at 1:32 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
Aron,
TestConnection is just used for lookup purpose only. (To list the resource while maintaining policies). There should be steps to create keytab for rangerlookup, just make sure that you have policy for that user so it can list the hdfs directories/files.
Even if the test connection fails it doesn’t stop you from maintaining policies and using ranger.
Regards,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, September 20, 2017 at 1:10 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
When I am in the Ranger Admin Service Manager -> Edit Service for HDFS, there is a Test Connection button. When I press it, it tries to login with Username and Password. We use keytabs. Tracing the ranger_admin.log, in BaseClient.java, the lookupPrincipal and lookupKeytab are not set. If I force these to be set in the code, it then uses a keytab.
I'm not certain how the keytabs are to be specified other than the install.properties file. Clearly I must not have specified them properly.
https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80
<https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80>
On Wed, Sep 20, 2017 at 3:00 PM, Ramesh Mani <rm...@hortonworks.com>> wrote:
Aron,
When you say Login in with user and password is that Ranger Admin UI login? Or is the hdfs plugin login into ranger to fetch the policy?
Looks like the NPE is not related to Ranger, but please check namenode.log what is there. Please enable debug on namenode and check it out.
Thanks,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, September 20, 2017 at 12:21 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
Thanks.
Having lots of issues trying to get this to work.
Issue 1 - Admin Server
I'm not exactly sure what I am doing right or wrong so far, but it is still trying to login with a user and password rather than a keytab. In BaseClient.login(), I hard-coded the keytab and principal, and then I see a proper HDFS file listing occurring. This however is failing (see issue 2). Looks like it is expecting some xalogin.xml that does not exist to set these properties. I still need to dig into why this does not exist.
I'm not clear what authentication mode means exactly. I don't think UNIX/LDAP/AD fit anything we do based on questioning the Hadoop team here. I'm not exactly clear what this setting is used for or which setting we should specify.
Issue 2 - Namenode
I installed the plugin and was able to restart the name node, but no policy data was in the cache directory, it appears unable to sync. When I do a "hadoop ls" on the command line, I get a NPE:
Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:238)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:243)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:182)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:499)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1605)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1623)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePath(FSDirectory.java:544)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getListingInt(FSDirStatAndListingOp.java:55)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:3695)
On Tue, Sep 19, 2017 at 2:53 PM, Ramesh Mani <rm...@hortonworks.com>> wrote:
Hi Aaron
Please check this out https://cwiki.apache.org/confluence/display/RANGER/Ranger+installation+in+Kerberized++Environment
Regards,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Tuesday, September 19, 2017 at 11:58 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: HDFS Kerberos documentation/setup
1) What documentation should I be following to install Ranger manually for a Kerberos Hadoop cluster? I am interested in the HDFS Plugin.
This is what I found linked from the apache site, but is very old:
https://cwiki.apache.org/confluence/display/RANGER/Ranger+Installation+Guide
2) Following those instructions, I see "Create a repository in Ranger Policy Manager. E.g. "local_hdfs"."
Is this the same as creating a Service? I see Services under HDFS on the Ranger admin server.
3) Creating an HDFS service lists a Username and Password. We don't use passwords for our clusters, but have keytabs. What should this mandatory field be? What is it used for?
4) How is this supposed to be setup for secure clusters? Is there any manually setup example I can be pointed to?
Thanks,
Aaron
Re: HDFS Kerberos documentation/setup
Posted by Ramesh Mani <rm...@hortonworks.com>.
Aron,
TestConnection is just used for lookup purpose only. (To list the resource while maintaining policies). There should be steps to create keytab for rangerlookup, just make sure that you have policy for that user so it can list the hdfs directories/files.
Even if the test connection fails it doesn’t stop you from maintaining policies and using ranger.
Regards,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, September 20, 2017 at 1:10 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
When I am in the Ranger Admin Service Manager -> Edit Service for HDFS, there is a Test Connection button. When I press it, it tries to login with Username and Password. We use keytabs. Tracing the ranger_admin.log, in BaseClient.java, the lookupPrincipal and lookupKeytab are not set. If I force these to be set in the code, it then uses a keytab.
I'm not certain how the keytabs are to be specified other than the install.properties file. Clearly I must not have specified them properly.
https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80
<https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80>
On Wed, Sep 20, 2017 at 3:00 PM, Ramesh Mani <rm...@hortonworks.com>> wrote:
Aron,
When you say Login in with user and password is that Ranger Admin UI login? Or is the hdfs plugin login into ranger to fetch the policy?
Looks like the NPE is not related to Ranger, but please check namenode.log what is there. Please enable debug on namenode and check it out.
Thanks,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, September 20, 2017 at 12:21 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
Thanks.
Having lots of issues trying to get this to work.
Issue 1 - Admin Server
I'm not exactly sure what I am doing right or wrong so far, but it is still trying to login with a user and password rather than a keytab. In BaseClient.login(), I hard-coded the keytab and principal, and then I see a proper HDFS file listing occurring. This however is failing (see issue 2). Looks like it is expecting some xalogin.xml that does not exist to set these properties. I still need to dig into why this does not exist.
I'm not clear what authentication mode means exactly. I don't think UNIX/LDAP/AD fit anything we do based on questioning the Hadoop team here. I'm not exactly clear what this setting is used for or which setting we should specify.
Issue 2 - Namenode
I installed the plugin and was able to restart the name node, but no policy data was in the cache directory, it appears unable to sync. When I do a "hadoop ls" on the command line, I get a NPE:
Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:238)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:243)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:182)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:499)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1605)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1623)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePath(FSDirectory.java:544)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getListingInt(FSDirStatAndListingOp.java:55)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:3695)
On Tue, Sep 19, 2017 at 2:53 PM, Ramesh Mani <rm...@hortonworks.com>> wrote:
Hi Aaron
Please check this out https://cwiki.apache.org/confluence/display/RANGER/Ranger+installation+in+Kerberized++Environment
Regards,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Tuesday, September 19, 2017 at 11:58 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: HDFS Kerberos documentation/setup
1) What documentation should I be following to install Ranger manually for a Kerberos Hadoop cluster? I am interested in the HDFS Plugin.
This is what I found linked from the apache site, but is very old:
https://cwiki.apache.org/confluence/display/RANGER/Ranger+Installation+Guide
2) Following those instructions, I see "Create a repository in Ranger Policy Manager. E.g. "local_hdfs"."
Is this the same as creating a Service? I see Services under HDFS on the Ranger admin server.
3) Creating an HDFS service lists a Username and Password. We don't use passwords for our clusters, but have keytabs. What should this mandatory field be? What is it used for?
4) How is this supposed to be setup for secure clusters? Is there any manually setup example I can be pointed to?
Thanks,
Aaron
Re: HDFS Kerberos documentation/setup
Posted by Aaron Gresch <ag...@gmail.com>.
When I am in the Ranger Admin Service Manager -> Edit Service for HDFS,
there is a Test Connection button. When I press it, it tries to login with
Username and Password. We use keytabs. Tracing the ranger_admin.log, in
BaseClient.java, the lookupPrincipal and lookupKeytab are not set. If I
force these to be set in the code, it then uses a keytab.
I'm not certain how the keytabs are to be specified other than the
install.properties file. Clearly I must not have specified them properly.
https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80
<https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80>
On Wed, Sep 20, 2017 at 3:00 PM, Ramesh Mani <rm...@hortonworks.com> wrote:
> Aron,
>
> When you say Login in with user and password is that Ranger Admin UI
> login? Or is the hdfs plugin login into ranger to fetch the policy?
>
> Looks like the NPE is not related to Ranger, but please check
> namenode.log what is there. Please enable debug on namenode and check it
> out.
>
> Thanks,
> Ramesh
>
> From: Aaron Gresch <ag...@gmail.com>
> Reply-To: "user@ranger.apache.org" <us...@ranger.apache.org>
> Date: Wednesday, September 20, 2017 at 12:21 PM
> To: "user@ranger.apache.org" <us...@ranger.apache.org>
> Subject: Re: HDFS Kerberos documentation/setup
>
>
> Thanks.
>
> Having lots of issues trying to get this to work.
>
> Issue 1 - Admin Server
>
> I'm not exactly sure what I am doing right or wrong so far, but it is
> still trying to login with a user and password rather than a keytab. In
> BaseClient.login(), I hard-coded the keytab and principal, and then I see a
> proper HDFS file listing occurring. This however is failing (see issue
> 2). Looks like it is expecting some xalogin.xml that does not exist to set
> these properties. I still need to dig into why this does not exist.
>
> I'm not clear what authentication mode means exactly. I don't think
> UNIX/LDAP/AD fit anything we do based on questioning the Hadoop team here.
> I'm not exactly clear what this setting is used for or which setting we
> should specify.
>
> Issue 2 - Namenode
>
> I installed the plugin and was able to restart the name node, but no
> policy data was in the cache directory, it appears unable to sync. When I
> do a "hadoop ls" on the command line, I get a NPE:
>
> Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
>
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:238)
>
> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> getINodeAttrs(FSPermissionChecker.java:243)
>
> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> checkPermission(FSPermissionChecker.java:182)
>
> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> checkTraverse(FSPermissionChecker.java:499)
>
> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> checkTraverse(FSDirectory.java:1605)
>
> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> checkTraverse(FSDirectory.java:1623)
>
> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> resolvePath(FSDirectory.java:544)
>
> at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.
> getListingInt(FSDirStatAndListingOp.java:55)
>
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getListing(FSNamesystem.java:3695)
>
>
> On Tue, Sep 19, 2017 at 2:53 PM, Ramesh Mani <rm...@hortonworks.com>
> wrote:
>
>> Hi Aaron
>>
>> Please check this out https://cwiki.apache.org/c
>> onfluence/display/RANGER/Ranger+installation+in+Kerberized++Environment
>>
>> Regards,
>> Ramesh
>>
>> From: Aaron Gresch <ag...@gmail.com>
>> Reply-To: "user@ranger.apache.org" <us...@ranger.apache.org>
>> Date: Tuesday, September 19, 2017 at 11:58 AM
>> To: "user@ranger.apache.org" <us...@ranger.apache.org>
>> Subject: HDFS Kerberos documentation/setup
>>
>> 1) What documentation should I be following to install Ranger manually
>> for a Kerberos Hadoop cluster? I am interested in the HDFS Plugin.
>>
>> This is what I found linked from the apache site, but is very old:
>>
>> https://cwiki.apache.org/confluence/display/RANGER/Ranger+In
>> stallation+Guide
>>
>>
>> 2) Following those instructions, I see "Create a repository in Ranger
>> Policy Manager. E.g. "local_hdfs"."
>>
>> Is this the same as creating a Service? I see Services under HDFS on the
>> Ranger admin server.
>>
>>
>> 3) Creating an HDFS service lists a Username and Password. We don't use
>> passwords for our clusters, but have keytabs. What should this mandatory
>> field be? What is it used for?
>>
>>
>> 4) How is this supposed to be setup for secure clusters? Is there any
>> manually setup example I can be pointed to?
>>
>>
>> Thanks,
>> Aaron
>>
>
>
Re: HDFS Kerberos documentation/setup
Posted by Ramesh Mani <rm...@hortonworks.com>.
Aron,
When you say Login in with user and password is that Ranger Admin UI login? Or is the hdfs plugin login into ranger to fetch the policy?
Looks like the NPE is not related to Ranger, but please check namenode.log what is there. Please enable debug on namenode and check it out.
Thanks,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, September 20, 2017 at 12:21 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: HDFS Kerberos documentation/setup
Thanks.
Having lots of issues trying to get this to work.
Issue 1 - Admin Server
I'm not exactly sure what I am doing right or wrong so far, but it is still trying to login with a user and password rather than a keytab. In BaseClient.login(), I hard-coded the keytab and principal, and then I see a proper HDFS file listing occurring. This however is failing (see issue 2). Looks like it is expecting some xalogin.xml that does not exist to set these properties. I still need to dig into why this does not exist.
I'm not clear what authentication mode means exactly. I don't think UNIX/LDAP/AD fit anything we do based on questioning the Hadoop team here. I'm not exactly clear what this setting is used for or which setting we should specify.
Issue 2 - Namenode
I installed the plugin and was able to restart the name node, but no policy data was in the cache directory, it appears unable to sync. When I do a "hadoop ls" on the command line, I get a NPE:
Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:238)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:243)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:182)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:499)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1605)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1623)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePath(FSDirectory.java:544)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getListingInt(FSDirStatAndListingOp.java:55)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:3695)
On Tue, Sep 19, 2017 at 2:53 PM, Ramesh Mani <rm...@hortonworks.com>> wrote:
Hi Aaron
Please check this out https://cwiki.apache.org/confluence/display/RANGER/Ranger+installation+in+Kerberized++Environment
Regards,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Tuesday, September 19, 2017 at 11:58 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: HDFS Kerberos documentation/setup
1) What documentation should I be following to install Ranger manually for a Kerberos Hadoop cluster? I am interested in the HDFS Plugin.
This is what I found linked from the apache site, but is very old:
https://cwiki.apache.org/confluence/display/RANGER/Ranger+Installation+Guide
2) Following those instructions, I see "Create a repository in Ranger Policy Manager. E.g. "local_hdfs"."
Is this the same as creating a Service? I see Services under HDFS on the Ranger admin server.
3) Creating an HDFS service lists a Username and Password. We don't use passwords for our clusters, but have keytabs. What should this mandatory field be? What is it used for?
4) How is this supposed to be setup for secure clusters? Is there any manually setup example I can be pointed to?
Thanks,
Aaron
Re: HDFS Kerberos documentation/setup
Posted by Aaron Gresch <ag...@gmail.com>.
Thanks.
Having lots of issues trying to get this to work.
Issue 1 - Admin Server
I'm not exactly sure what I am doing right or wrong so far, but it is still
trying to login with a user and password rather than a keytab. In
BaseClient.login(), I hard-coded the keytab and principal, and then I see a
proper HDFS file listing occurring. This however is failing (see issue
2). Looks like it is expecting some xalogin.xml that does not exist to set
these properties. I still need to dig into why this does not exist.
I'm not clear what authentication mode means exactly. I don't think
UNIX/LDAP/AD fit anything we do based on questioning the Hadoop team here.
I'm not exactly clear what this setting is used for or which setting we
should specify.
Issue 2 - Namenode
I installed the plugin and was able to restart the name node, but no policy
data was in the cache directory, it appears unable to sync. When I do a
"hadoop ls" on the command line, I get a NPE:
Caused by:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:238)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:243)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:182)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:499)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1605)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1623)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePath(FSDirectory.java:544)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getListingInt(FSDirStatAndListingOp.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:3695)
On Tue, Sep 19, 2017 at 2:53 PM, Ramesh Mani <rm...@hortonworks.com> wrote:
> Hi Aaron
>
> Please check this out https://cwiki.apache.org/confluence/display/RANGER/
> Ranger+installation+in+Kerberized++Environment
>
> Regards,
> Ramesh
>
> From: Aaron Gresch <ag...@gmail.com>
> Reply-To: "user@ranger.apache.org" <us...@ranger.apache.org>
> Date: Tuesday, September 19, 2017 at 11:58 AM
> To: "user@ranger.apache.org" <us...@ranger.apache.org>
> Subject: HDFS Kerberos documentation/setup
>
> 1) What documentation should I be following to install Ranger manually for
> a Kerberos Hadoop cluster? I am interested in the HDFS Plugin.
>
> This is what I found linked from the apache site, but is very old:
>
> https://cwiki.apache.org/confluence/display/RANGER/Ranger+
> Installation+Guide
>
>
> 2) Following those instructions, I see "Create a repository in Ranger
> Policy Manager. E.g. "local_hdfs"."
>
> Is this the same as creating a Service? I see Services under HDFS on the
> Ranger admin server.
>
>
> 3) Creating an HDFS service lists a Username and Password. We don't use
> passwords for our clusters, but have keytabs. What should this mandatory
> field be? What is it used for?
>
>
> 4) How is this supposed to be setup for secure clusters? Is there any
> manually setup example I can be pointed to?
>
>
> Thanks,
> Aaron
>
Re: HDFS Kerberos documentation/setup
Posted by Ramesh Mani <rm...@hortonworks.com>.
Hi Aaron
Please check this out https://cwiki.apache.org/confluence/display/RANGER/Ranger+installation+in+Kerberized++Environment
Regards,
Ramesh
From: Aaron Gresch <ag...@gmail.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Tuesday, September 19, 2017 at 11:58 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: HDFS Kerberos documentation/setup
1) What documentation should I be following to install Ranger manually for a Kerberos Hadoop cluster? I am interested in the HDFS Plugin.
This is what I found linked from the apache site, but is very old:
https://cwiki.apache.org/confluence/display/RANGER/Ranger+Installation+Guide
2) Following those instructions, I see "Create a repository in Ranger Policy Manager. E.g. "local_hdfs"."
Is this the same as creating a Service? I see Services under HDFS on the Ranger admin server.
3) Creating an HDFS service lists a Username and Password. We don't use passwords for our clusters, but have keytabs. What should this mandatory field be? What is it used for?
4) How is this supposed to be setup for secure clusters? Is there any manually setup example I can be pointed to?
Thanks,
Aaron