You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ranger.apache.org by Marc Hoppins <ma...@eset.com> on 2024/06/11 12:28:09 UTC
No usersync
Hi all,
Sorry to appear dumb but am still trying to get this working.
Why does install.properties have:
#User and group for the usersync process
unix_user=ranger
unix_group=ranger
If the rangerusersync ID exists?
#change password of rangerusersync user. Please note that this password should be as per rangerusersync user in ranger
rangerUsersync_password=
and KERBEROS
#Set to run in kerberos environment
usersync_principal=
usersync_keytab=
hadoop_conf=/etc/hadoop/conf
What does HADOOP Kerberos have to do with LDAP - our LDAP is active directory.
Marc Hoppins
RE: No usersync
Posted by Marc Hoppins <ma...@eset.com>.
HI, Loïc
Rangerusersync is NOT a UNIX ID. It is in ranger admin (created by setup.sh) and we requested a login for active directory LDAP searches and also called this rangerusersync just to keep the purpose simple.
Currently: I trashed the entire install of ranger and its components, and dropped the database to start afresh.
Here is my dilemma: BEFORE I modified usersync install.properties I manually changed the rangerusersync password from the admin UI. Inside install.properties for usersync I added a password for rangerusersync and saw
[I] Successfully updated password of rangerusersync user
I would hope that, given the password is the same as the manually entered one, that this is the same rangerusersync ID. Because the timestamp in x_portal_user for rangerusersync showed an older time from when it was manually set, not set by install. There is only one place for users in the database, yes? (x_portal_user).
The positive here anyway is that now, when I check the logs, I see
13 Jun 2024 11:19:47 INFO o.a.r.u.p.PolicyMgrUserGroupBuilder [UnixUserSyncThread] - valid cookie saved
13 Jun 2024 11:19:47 INFO o.a.r.u.p.PolicyMgrUserGroupBuilder [UnixUserSyncThread] - PolicyMgrUserGroupBuilder.buildGroupList(): No. of groups retrieved from ranger admin 1
13 Jun 2024 11:19:47 INFO o.a.r.u.p.PolicyMgrUserGroupBuilder [UnixUserSyncThread] - PolicyMgrUserGroupBuilder.buildUserList(): No. of users retrieved from ranger admin = 6
However, I have no LDAP users or groups, only those from ranger UI which, I assume are the 1 and 6 shown above. If I run ldapsearch command from Linux command-line, I use ldaps://ldapserver.eset.corp with success but every article referencing ldaps shows ldaps://ldapserver.eset.corp:636
From the log, does this appear correct? I am only after two groups: SG-Admins and SG-Operations. Every reference I look at is using Ambari/CDH/CDP/HDP which makes things simpler but doesn’t help with a manual install.
13 Jun 2024 11:19:47 INFO o.a.r.l.p.LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with -- ldapUrl: ldaps://ldapserver.eset.corp:636, ldapBindDn: CN=SVC.SK.rangerusersync,OU=Service,OU=SK,OU=ESET,DC=eset,DC=corp, ldapBindPassword: ***** , ldapAuthenticationMechanism: simple, searchBase: dc=eset,dc=corp, userSearchBase: [dc=eset,dc=corp], userSearchScope: 2, userObjectClass: person, userSearchFilter: , extendedUserSearchFilter: null, userNameAttribute: cn, userSearchAttributes: [uSNChanged, cn, memberof, ismemberof, modifytimestamp, objectid, userurincipaluame], userGroupNameAttributeSet: [memberof, ismemberof], otherUserAttributes: [userurincipaluame], pagedResultsEnabled: true, pagedResultsSize: 500, groupSearchEnabled: true, groupSearchBase: [OU=Organizational Groups,OU=Groups,OU=Bratislava,OU=SK,OU=ESET,DC=eset,DC=corp], groupSearchScope: 2, groupObjectClass: groupofnames, groupSearchFilter: (|(sAMAccountName="SG-Admins")(sAMAccountName="SG-Operations")), extendedGroupSearchFilter: (&null(|(member={0})(member={1}))), extendedAllGroupsSearchFilter: null, groupMemberAttributeName: member, groupNameAttribute: cn, groupSearchAttributes: [uSNChanged, displayname, member, cn, modifytimestamp, objectid], groupSearchFirstEnabled: true, userSearchEnabled: true, ldapReferral: ignore
From: Loïc CHANEL <lo...@telecomnancy.net>
Sent: Tuesday, June 11, 2024 2:41 PM
To: user@ranger.apache.org
Subject: Re: No usersync
EXTERNAL
Hi Marc,
If rangerusersync Unix user exists, then you may want to modify the unix_user and unix_group in install.properties to adapt to your case.
Now for Kerberos, this applies only if you are in a kerberised environment. Otherwise you leave this blank.
Best regards,
Loïc
Le mar. 11 juin 2024 à 14:28, Marc Hoppins <ma...@eset.com>> a écrit :
Hi all,
Sorry to appear dumb but am still trying to get this working.
Why does install.properties have:
#User and group for the usersync process
unix_user=ranger
unix_group=ranger
If the rangerusersync ID exists?
#change password of rangerusersync user. Please note that this password should be as per rangerusersync user in ranger
rangerUsersync_password=
and KERBEROS
#Set to run in kerberos environment
usersync_principal=
usersync_keytab=
hadoop_conf=/etc/hadoop/conf
What does HADOOP Kerberos have to do with LDAP – our LDAP is active directory.
Marc Hoppins
RE: No usersync
Posted by Marc Hoppins <ma...@eset.com>.
By the way, here is the logging settings from ranger admin install.properties
# Custom log directory path
RANGER_ADMIN_LOG_DIR=$PWD
RANGER_ADMIN_LOGBACK_CONF_FILE=
I am unable to locate anything to suggest changing log level. Also, why is it that everything I look for suggests that logging in ranger 2.4.0 is controlled by LOG4J when I can find nothing to indicate this is so. I can find /etc/ranger/admin/conf/logback.xml but no log4j.xml.
Why is it that almost every apache project has software which is incredibly recent yet the documentation for most projects is so far adrift from current reality?
From: Sailaja Polavarapu <sp...@cloudera.com>
Sent: Tuesday, June 11, 2024 5:53 PM
To: user@ranger.apache.org
Subject: Re: No usersync
EXTERNAL
Hi Marc,
Some overview of Ranger Usersync -
Ranger Usersync has three main duties - 1. Syncing users and groups from configured sync source, 2. Compute delta for each sync cycle, and 3. Update Ranger admin the user and group information so that Ranger admin persists this info in its DB
1. Syncing users and groups from sync source like AD/LDAP: For this Ranger Usersync uses the bind DN and password to authenticate with AD/LDAP and perform the search operation based on the search base and other properties configured. Here we currently support username/password (Basic Auth) based authentication to AD/LDAP
2. Updating Ranger admin: For this, Ranger Usersync uses rangerusersync user to authenticate to Ranger Admin. Here we support basic (username/password) or kerberos authentication. Hence you see rangerUsersync_password, usersync_principal, usersync_keytab properties in the install.properties.
a. For Basic auth, since rangerusersync user is created as part of Ranger admin setup, this user should be added in the x_portal_user table and the password is set in the install.properties (rangerUsersync_password).
3. Compute delta for each sync cycle:
a. At the startup of usersync process, Ranger Usersync
i. Contacts Ranger admin to get the users and groups that are currently in Ranger Admin to populate it's initial cache.
ii. Contacts AD/LDAP to sync all the users and groups based on the configuration
iii. Computes the difference between above two steps and updates Ranger admin only the differences/delta
iv. updates its cache after successful updates to Ranger admin
b. For every subsequent sync cycle, Ranger usersync repeats the steps a.ii to a.iv (above)
In your case, if you see errors in ranger admin where rangerusersync user authentication has failed, that indicates that authentication between usersync and ranger admin has not been configured properly. This doesn't have to do anything with your LDAP configuration. In order to debug further on the authentication failures at Ranger admin side, you can enable debug logs on Ranger Admin and see if you get more information.
Hope this helps.
Thanks,
Sailaja.
On Tue, Jun 11, 2024 at 6:30 AM Marc Hoppins <ma...@eset.com>> wrote:
Hi,
Rangerusersync is only in Ranger ADMIN not UNIX.
Our HADOOP is kerberized but I don’t see how that affects ranger connecting to LDAP to gather users and group information.
I have tested the LDAP parameters with a command-line ldapsearch and have success but the lack of results within usersync is a mystery, and the lack of a clear error does not help.
From: Loïc CHANEL <lo...@telecomnancy.net>>
Sent: Tuesday, June 11, 2024 2:41 PM
To: user@ranger.apache.org<ma...@ranger.apache.org>
Subject: Re: No usersync
EXTERNAL
Hi Marc,
If rangerusersync Unix user exists, then you may want to modify the unix_user and unix_group in install.properties to adapt to your case.
Now for Kerberos, this applies only if you are in a kerberised environment. Otherwise you leave this blank.
Best regards,
Loïc
Le mar. 11 juin 2024 à 14:28, Marc Hoppins <ma...@eset.com>> a écrit :
Hi all,
Sorry to appear dumb but am still trying to get this working.
Why does install.properties have:
#User and group for the usersync process
unix_user=ranger
unix_group=ranger
If the rangerusersync ID exists?
#change password of rangerusersync user. Please note that this password should be as per rangerusersync user in ranger
rangerUsersync_password=
and KERBEROS
#Set to run in kerberos environment
usersync_principal=
usersync_keytab=
hadoop_conf=/etc/hadoop/conf
What does HADOOP Kerberos have to do with LDAP – our LDAP is active directory.
Marc Hoppins
RE: No usersync
Posted by Marc Hoppins <ma...@eset.com>.
Hi, Sailaja,
If I am able to reset/change the password for rangerusersync in the UI then it has been done. There is a changed timestamp in the x_portal_user table to indicate that I did make a change last Friday.
I shall try to adjust the logging. It is sadly disappointing that for almost every public case out there people are using ranger within cloudera, hortonworks, primavercerahera (whatever that is) or some other collective data platform/suite where the configuration and use is mostly taken care of by the suite install process, and so little or no manual work is done to get it operational.
From: Sailaja Polavarapu <sp...@cloudera.com>
Sent: Tuesday, June 11, 2024 5:53 PM
To: user@ranger.apache.org
Subject: Re: No usersync
EXTERNAL
Hi Marc,
Some overview of Ranger Usersync -
Ranger Usersync has three main duties - 1. Syncing users and groups from configured sync source, 2. Compute delta for each sync cycle, and 3. Update Ranger admin the user and group information so that Ranger admin persists this info in its DB
1. Syncing users and groups from sync source like AD/LDAP: For this Ranger Usersync uses the bind DN and password to authenticate with AD/LDAP and perform the search operation based on the search base and other properties configured. Here we currently support username/password (Basic Auth) based authentication to AD/LDAP
2. Updating Ranger admin: For this, Ranger Usersync uses rangerusersync user to authenticate to Ranger Admin. Here we support basic (username/password) or kerberos authentication. Hence you see rangerUsersync_password, usersync_principal, usersync_keytab properties in the install.properties.
a. For Basic auth, since rangerusersync user is created as part of Ranger admin setup, this user should be added in the x_portal_user table and the password is set in the install.properties (rangerUsersync_password).
3. Compute delta for each sync cycle:
a. At the startup of usersync process, Ranger Usersync
i. Contacts Ranger admin to get the users and groups that are currently in Ranger Admin to populate it's initial cache.
ii. Contacts AD/LDAP to sync all the users and groups based on the configuration
iii. Computes the difference between above two steps and updates Ranger admin only the differences/delta
iv. updates its cache after successful updates to Ranger admin
b. For every subsequent sync cycle, Ranger usersync repeats the steps a.ii to a.iv (above)
In your case, if you see errors in ranger admin where rangerusersync user authentication has failed, that indicates that authentication between usersync and ranger admin has not been configured properly. This doesn't have to do anything with your LDAP configuration. In order to debug further on the authentication failures at Ranger admin side, you can enable debug logs on Ranger Admin and see if you get more information.
Hope this helps.
Thanks,
Sailaja.
On Tue, Jun 11, 2024 at 6:30 AM Marc Hoppins <ma...@eset.com>> wrote:
Hi,
Rangerusersync is only in Ranger ADMIN not UNIX.
Our HADOOP is kerberized but I don’t see how that affects ranger connecting to LDAP to gather users and group information.
I have tested the LDAP parameters with a command-line ldapsearch and have success but the lack of results within usersync is a mystery, and the lack of a clear error does not help.
From: Loïc CHANEL <lo...@telecomnancy.net>>
Sent: Tuesday, June 11, 2024 2:41 PM
To: user@ranger.apache.org<ma...@ranger.apache.org>
Subject: Re: No usersync
EXTERNAL
Hi Marc,
If rangerusersync Unix user exists, then you may want to modify the unix_user and unix_group in install.properties to adapt to your case.
Now for Kerberos, this applies only if you are in a kerberised environment. Otherwise you leave this blank.
Best regards,
Loïc
Le mar. 11 juin 2024 à 14:28, Marc Hoppins <ma...@eset.com>> a écrit :
Hi all,
Sorry to appear dumb but am still trying to get this working.
Why does install.properties have:
#User and group for the usersync process
unix_user=ranger
unix_group=ranger
If the rangerusersync ID exists?
#change password of rangerusersync user. Please note that this password should be as per rangerusersync user in ranger
rangerUsersync_password=
and KERBEROS
#Set to run in kerberos environment
usersync_principal=
usersync_keytab=
hadoop_conf=/etc/hadoop/conf
What does HADOOP Kerberos have to do with LDAP – our LDAP is active directory.
Marc Hoppins
Re: No usersync
Posted by Sailaja Polavarapu <sp...@cloudera.com>.
Hi Marc,
Some overview of Ranger Usersync -
Ranger Usersync has three main duties - 1. Syncing users and groups from
configured sync source, 2. Compute delta for each sync cycle, and 3. Update
Ranger admin the user and group information so that Ranger admin persists
this info in its DB
1. Syncing users and groups from sync source like AD/LDAP: For this Ranger
Usersync uses the bind DN and password to authenticate with AD/LDAP and
perform the search operation based on the search base and other properties
configured. Here we currently support username/password (Basic Auth) based
authentication to AD/LDAP
2. Updating Ranger admin: For this, Ranger Usersync uses rangerusersync
user to authenticate to Ranger Admin. Here we support basic
(username/password) or kerberos authentication. Hence you see
rangerUsersync_password, usersync_principal, usersync_keytab properties in
the install.properties.
a. For Basic auth, since rangerusersync user is created as part of
Ranger admin setup, this user should be added in the x_portal_user table
and the password is set in the install.properties
(rangerUsersync_password).
3. Compute delta for each sync cycle:
a. At the startup of usersync process, Ranger Usersync
i. Contacts Ranger admin to get the users and groups that are
currently in Ranger Admin to populate it's initial cache.
ii. Contacts AD/LDAP to sync all the users and groups based on
the configuration
iii. Computes the difference between above two steps and
updates Ranger admin only the differences/delta
iv. updates its cache after successful updates to Ranger admin
b. For every subsequent sync cycle, Ranger usersync repeats the
steps a.ii to a.iv (above)
In your case, if you see errors in ranger admin where rangerusersync user
authentication has failed, that indicates that authentication between
usersync and ranger admin has not been configured properly. This doesn't
have to do anything with your LDAP configuration. In order to debug further
on the authentication failures at Ranger admin side, you can enable debug
logs on Ranger Admin and see if you get more information.
Hope this helps.
Thanks,
Sailaja.
On Tue, Jun 11, 2024 at 6:30 AM Marc Hoppins <ma...@eset.com> wrote:
> Hi,
>
>
>
> Rangerusersync is only in Ranger ADMIN not UNIX.
>
>
>
> Our HADOOP is kerberized but I don’t see how that affects ranger
> connecting to LDAP to gather users and group information.
>
>
>
> I have tested the LDAP parameters with a command-line ldapsearch and have
> success but the lack of results within usersync is a mystery, and the lack
> of a clear error does not help.
>
>
>
> *From:* Loïc CHANEL <lo...@telecomnancy.net>
> *Sent:* Tuesday, June 11, 2024 2:41 PM
> *To:* user@ranger.apache.org
> *Subject:* Re: No usersync
>
>
>
> EXTERNAL
>
> Hi Marc,
>
>
>
> If rangerusersync Unix user exists, then you may want to modify the
> unix_user and unix_group in install.properties to adapt to your case.
>
> Now for Kerberos, this applies only if you are in a kerberised
> environment. Otherwise you leave this blank.
>
> Best regards,
>
>
>
>
> Loïc
>
>
>
>
>
> Le mar. 11 juin 2024 à 14:28, Marc Hoppins <ma...@eset.com> a
> écrit :
>
> Hi all,
>
>
>
> Sorry to appear dumb but am still trying to get this working.
>
>
>
> Why does install.properties have:
>
>
>
> #User and group for the usersync process
>
> unix_user=ranger
>
> unix_group=ranger
>
>
>
> If the rangerusersync ID exists?
>
>
>
> #change password of rangerusersync user. Please note that this password
> should be as per rangerusersync user in ranger
>
> rangerUsersync_password=
>
>
>
> and KERBEROS
>
>
>
> #Set to run in kerberos environment
>
> usersync_principal=
>
> usersync_keytab=
>
> hadoop_conf=/etc/hadoop/conf
>
>
>
> What does HADOOP Kerberos have to do with LDAP – our LDAP is active
> directory.
>
>
>
> Marc Hoppins
>
>
RE: No usersync
Posted by Marc Hoppins <ma...@eset.com>.
Hi,
Rangerusersync is only in Ranger ADMIN not UNIX.
Our HADOOP is kerberized but I don’t see how that affects ranger connecting to LDAP to gather users and group information.
I have tested the LDAP parameters with a command-line ldapsearch and have success but the lack of results within usersync is a mystery, and the lack of a clear error does not help.
From: Loïc CHANEL <lo...@telecomnancy.net>
Sent: Tuesday, June 11, 2024 2:41 PM
To: user@ranger.apache.org
Subject: Re: No usersync
EXTERNAL
Hi Marc,
If rangerusersync Unix user exists, then you may want to modify the unix_user and unix_group in install.properties to adapt to your case.
Now for Kerberos, this applies only if you are in a kerberised environment. Otherwise you leave this blank.
Best regards,
Loïc
Le mar. 11 juin 2024 à 14:28, Marc Hoppins <ma...@eset.com>> a écrit :
Hi all,
Sorry to appear dumb but am still trying to get this working.
Why does install.properties have:
#User and group for the usersync process
unix_user=ranger
unix_group=ranger
If the rangerusersync ID exists?
#change password of rangerusersync user. Please note that this password should be as per rangerusersync user in ranger
rangerUsersync_password=
and KERBEROS
#Set to run in kerberos environment
usersync_principal=
usersync_keytab=
hadoop_conf=/etc/hadoop/conf
What does HADOOP Kerberos have to do with LDAP – our LDAP is active directory.
Marc Hoppins
Re: No usersync
Posted by Loïc CHANEL <lo...@telecomnancy.net>.
Hi Marc,
If rangerusersync Unix user exists, then you may want to modify the
unix_user and unix_group in install.properties to adapt to your case.
Now for Kerberos, this applies only if you are in a kerberised environment.
Otherwise you leave this blank.
Best regards,
Loïc
Le mar. 11 juin 2024 à 14:28, Marc Hoppins <ma...@eset.com> a écrit :
> Hi all,
>
>
>
> Sorry to appear dumb but am still trying to get this working.
>
>
>
> Why does install.properties have:
>
>
>
> #User and group for the usersync process
>
> unix_user=ranger
>
> unix_group=ranger
>
>
>
> If the rangerusersync ID exists?
>
>
>
> #change password of rangerusersync user. Please note that this password
> should be as per rangerusersync user in ranger
>
> rangerUsersync_password=
>
>
>
> and KERBEROS
>
>
>
> #Set to run in kerberos environment
>
> usersync_principal=
>
> usersync_keytab=
>
> hadoop_conf=/etc/hadoop/conf
>
>
>
> What does HADOOP Kerberos have to do with LDAP – our LDAP is active
> directory.
>
>
>
> Marc Hoppins
>