You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Emil Anca <ea...@hortonworks.com> on 2015/04/17 14:12:23 UTC
Review Request 33302: ambari-server sync-ldap fails if there are too
many users in the LDAP server (more than 1000?)
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33302/
-----------------------------------------------------------
Review request for Ambari, Robert Levas and Tom Beerbower.
Bugs: AMBARI-10513
https://issues.apache.org/jira/browse/AMBARI-10513
Repository: ambari
Description
-------
ambari-server setup-ldap was performed against QE AD server, which has more than 2000 users.
[root@c6401 ~]# ambari-server sync-ldap --all
Using python /usr/bin/python2.6
Syncing with LDAP...
Enter Ambari Admin login: admin
Enter Ambari Admin password:
Syncing all.......ERROR: Exiting with exit code 1.
REASON: Caught exception running LDAP sync. [LDAP: error code 4 - Sizelimit Exceeded]; nested exception is javax.naming.SizeLimitExceededException: [LDAP: error code 4 - Sizelimit Exceeded]; remaining name 'CN=Users,DC=scl42,DC=hortonworks,DC=com'
Dilli Arumugam told me that Ranger (formerly known as XASecure) also hit a similar issue: BUG-23578.
BUG-31217: Attempting to sync LDAP that contains > 1000 entities fails.
Problem: If your directory contains > 1000 users, attempts to sync-ldap users and groups to Ambari will fail. There is a limit of 1000 to the number of entities Ambari can process.
Solution:
The solution is to retrieve the results in batches by using result paging.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulator.java d1293cb
ambari-server/src/test/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulatorTest.java 09a2256
Diff: https://reviews.apache.org/r/33302/diff/
Testing
-------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [3.293s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.043s]
[INFO] Ambari Web ........................................ SUCCESS [42.741s]
[INFO] Ambari Views ...................................... SUCCESS [3.783s]
[INFO] Ambari Admin View ................................. SUCCESS [12.679s]
[INFO] Ambari Metrics Common ............................. SUCCESS [1.661s]
[INFO] Ambari Server ..................................... SUCCESS [35:50.606s]
[INFO] Ambari Agent ...................................... SUCCESS [10.798s]
[INFO] Ambari Client ..................................... SUCCESS [0.041s]
[INFO] Ambari Python Client .............................. SUCCESS [0.246s]
[INFO] Ambari Groovy Client .............................. SUCCESS [10.219s]
[INFO] Ambari Shell ...................................... SUCCESS [0.056s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.038s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [6.721s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 37:23.499s
[INFO] Finished at: Fri Apr 17 13:02:14 EEST 2015
[INFO] Final Memory: 68M/565M
[INFO] ------------------------------------------------------------------------
Thanks,
Emil Anca
Re: Review Request 33302: ambari-server sync-ldap fails if there are
too many users in the LDAP server (more than 1000?)
Posted by Emil Anca <ea...@hortonworks.com>.
> On April 17, 2015, 1:31 p.m., Robert Levas wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulator.java, line 639
> > <https://reviews.apache.org/r/33302/diff/1/?file=933109#file933109line639>
> >
> > According to the docs, It is required that a relevant `PagedResultsCookie` be passed in for multiple requests? Is this correct?
For the initial request, the PagedResultsDirContextProcessor is to be initialez with a null PagedResultsCookie or using a different constructor. This PagedResultsDirContextProcessor is then passed to the LdapTemplate#search method which will update the PagedResultsCookie as needed based on what is sent from the LDAP Server. As a result, sequential calls will use the updated cookie and there is no need to pass it again.
> On April 17, 2015, 1:31 p.m., Robert Levas wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulator.java, line 533
> > <https://reviews.apache.org/r/33302/diff/1/?file=933109#file933109line533>
> >
> > I am not familair with the `PagedResultsDirContextProcessor`, but looking at the docs it seems like a `NullPointerException` may be thrown since it is possible that `processor.getCookie()` will return `null`.
The LdapTemplate#search which uses the PagedResultsDirContextProcessor is responsible for updating the cookie based on what is sent back from the LDAP Server and hence processor.getCookie() *should* never be null in this context. The usage docs of the LdapTemplate with a processor do not use a nullcheck and therefore I am assuming that the spring wrapper will always instantiate PagedResultsDirContextProcessor.PagedResultsCookie.
- Emil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33302/#review80450
-----------------------------------------------------------
On April 17, 2015, 12:12 p.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33302/
> -----------------------------------------------------------
>
> (Updated April 17, 2015, 12:12 p.m.)
>
>
> Review request for Ambari, Robert Levas and Tom Beerbower.
>
>
> Bugs: AMBARI-10513
> https://issues.apache.org/jira/browse/AMBARI-10513
>
>
> Repository: ambari
>
>
> Description
> -------
>
> ambari-server setup-ldap was performed against QE AD server, which has more than 2000 users.
> [root@c6401 ~]# ambari-server sync-ldap --all
> Using python /usr/bin/python2.6
> Syncing with LDAP...
> Enter Ambari Admin login: admin
> Enter Ambari Admin password:
> Syncing all.......ERROR: Exiting with exit code 1.
> REASON: Caught exception running LDAP sync. [LDAP: error code 4 - Sizelimit Exceeded]; nested exception is javax.naming.SizeLimitExceededException: [LDAP: error code 4 - Sizelimit Exceeded]; remaining name 'CN=Users,DC=scl42,DC=hortonworks,DC=com'
> Dilli Arumugam told me that Ranger (formerly known as XASecure) also hit a similar issue: BUG-23578.
> BUG-31217: Attempting to sync LDAP that contains > 1000 entities fails.
> Problem: If your directory contains > 1000 users, attempts to sync-ldap users and groups to Ambari will fail. There is a limit of 1000 to the number of entities Ambari can process.
>
>
> Solution:
>
> The solution is to retrieve the results in batches by using result paging.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulator.java d1293cb
> ambari-server/src/test/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulatorTest.java 09a2256
>
> Diff: https://reviews.apache.org/r/33302/diff/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [3.293s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.043s]
> [INFO] Ambari Web ........................................ SUCCESS [42.741s]
> [INFO] Ambari Views ...................................... SUCCESS [3.783s]
> [INFO] Ambari Admin View ................................. SUCCESS [12.679s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.661s]
> [INFO] Ambari Server ..................................... SUCCESS [35:50.606s]
> [INFO] Ambari Agent ...................................... SUCCESS [10.798s]
> [INFO] Ambari Client ..................................... SUCCESS [0.041s]
> [INFO] Ambari Python Client .............................. SUCCESS [0.246s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [10.219s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.056s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.038s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [6.721s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 37:23.499s
> [INFO] Finished at: Fri Apr 17 13:02:14 EEST 2015
> [INFO] Final Memory: 68M/565M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Emil Anca
>
>
Re: Review Request 33302: ambari-server sync-ldap fails if there are
too many users in the LDAP server (more than 1000?)
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33302/#review80450
-----------------------------------------------------------
Ship it!
ambari-server/src/main/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulator.java
<https://reviews.apache.org/r/33302/#comment130365>
I am not familair with the `PagedResultsDirContextProcessor`, but looking at the docs it seems like a `NullPointerException` may be thrown since it is possible that `processor.getCookie()` will return `null`.
ambari-server/src/main/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulator.java
<https://reviews.apache.org/r/33302/#comment130367>
According to the docs, It is required that a relevant `PagedResultsCookie` be passed in for multiple requests? Is this correct?
- Robert Levas
On April 17, 2015, 8:12 a.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33302/
> -----------------------------------------------------------
>
> (Updated April 17, 2015, 8:12 a.m.)
>
>
> Review request for Ambari, Robert Levas and Tom Beerbower.
>
>
> Bugs: AMBARI-10513
> https://issues.apache.org/jira/browse/AMBARI-10513
>
>
> Repository: ambari
>
>
> Description
> -------
>
> ambari-server setup-ldap was performed against QE AD server, which has more than 2000 users.
> [root@c6401 ~]# ambari-server sync-ldap --all
> Using python /usr/bin/python2.6
> Syncing with LDAP...
> Enter Ambari Admin login: admin
> Enter Ambari Admin password:
> Syncing all.......ERROR: Exiting with exit code 1.
> REASON: Caught exception running LDAP sync. [LDAP: error code 4 - Sizelimit Exceeded]; nested exception is javax.naming.SizeLimitExceededException: [LDAP: error code 4 - Sizelimit Exceeded]; remaining name 'CN=Users,DC=scl42,DC=hortonworks,DC=com'
> Dilli Arumugam told me that Ranger (formerly known as XASecure) also hit a similar issue: BUG-23578.
> BUG-31217: Attempting to sync LDAP that contains > 1000 entities fails.
> Problem: If your directory contains > 1000 users, attempts to sync-ldap users and groups to Ambari will fail. There is a limit of 1000 to the number of entities Ambari can process.
>
>
> Solution:
>
> The solution is to retrieve the results in batches by using result paging.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulator.java d1293cb
> ambari-server/src/test/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulatorTest.java 09a2256
>
> Diff: https://reviews.apache.org/r/33302/diff/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [3.293s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.043s]
> [INFO] Ambari Web ........................................ SUCCESS [42.741s]
> [INFO] Ambari Views ...................................... SUCCESS [3.783s]
> [INFO] Ambari Admin View ................................. SUCCESS [12.679s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.661s]
> [INFO] Ambari Server ..................................... SUCCESS [35:50.606s]
> [INFO] Ambari Agent ...................................... SUCCESS [10.798s]
> [INFO] Ambari Client ..................................... SUCCESS [0.041s]
> [INFO] Ambari Python Client .............................. SUCCESS [0.246s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [10.219s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.056s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.038s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [6.721s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 37:23.499s
> [INFO] Finished at: Fri Apr 17 13:02:14 EEST 2015
> [INFO] Final Memory: 68M/565M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Emil Anca
>
>
Re: Review Request 33302: ambari-server sync-ldap fails if there are
too many users in the LDAP server (more than 1000?)
Posted by Tom Beerbower <tb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33302/#review80706
-----------------------------------------------------------
Ship it!
Looks good. Thanks!
- Tom Beerbower
On April 17, 2015, 12:12 p.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33302/
> -----------------------------------------------------------
>
> (Updated April 17, 2015, 12:12 p.m.)
>
>
> Review request for Ambari, Robert Levas and Tom Beerbower.
>
>
> Bugs: AMBARI-10513
> https://issues.apache.org/jira/browse/AMBARI-10513
>
>
> Repository: ambari
>
>
> Description
> -------
>
> ambari-server setup-ldap was performed against QE AD server, which has more than 2000 users.
> [root@c6401 ~]# ambari-server sync-ldap --all
> Using python /usr/bin/python2.6
> Syncing with LDAP...
> Enter Ambari Admin login: admin
> Enter Ambari Admin password:
> Syncing all.......ERROR: Exiting with exit code 1.
> REASON: Caught exception running LDAP sync. [LDAP: error code 4 - Sizelimit Exceeded]; nested exception is javax.naming.SizeLimitExceededException: [LDAP: error code 4 - Sizelimit Exceeded]; remaining name 'CN=Users,DC=scl42,DC=hortonworks,DC=com'
> Dilli Arumugam told me that Ranger (formerly known as XASecure) also hit a similar issue: BUG-23578.
> BUG-31217: Attempting to sync LDAP that contains > 1000 entities fails.
> Problem: If your directory contains > 1000 users, attempts to sync-ldap users and groups to Ambari will fail. There is a limit of 1000 to the number of entities Ambari can process.
>
>
> Solution:
>
> The solution is to retrieve the results in batches by using result paging.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulator.java d1293cb
> ambari-server/src/test/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulatorTest.java 09a2256
>
> Diff: https://reviews.apache.org/r/33302/diff/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [3.293s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.043s]
> [INFO] Ambari Web ........................................ SUCCESS [42.741s]
> [INFO] Ambari Views ...................................... SUCCESS [3.783s]
> [INFO] Ambari Admin View ................................. SUCCESS [12.679s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.661s]
> [INFO] Ambari Server ..................................... SUCCESS [35:50.606s]
> [INFO] Ambari Agent ...................................... SUCCESS [10.798s]
> [INFO] Ambari Client ..................................... SUCCESS [0.041s]
> [INFO] Ambari Python Client .............................. SUCCESS [0.246s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [10.219s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.056s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.038s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [6.721s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 37:23.499s
> [INFO] Finished at: Fri Apr 17 13:02:14 EEST 2015
> [INFO] Final Memory: 68M/565M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Emil Anca
>
>