You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ambari.apache.org by Gautam Borad <gb...@gmail.com> on 2016/01/18 17:58:44 UTC

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42452/
-----------------------------------------------------------

(Updated Jan. 18, 2016, 4:58 p.m.)


Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.


Summary (updated)
-----------------

AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger


Bugs: AMBARI-14170
    https://issues.apache.org/jira/browse/AMBARI-14170


Repository: ambari


Description (updated)
-------

Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.

This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.


Diffs (updated)
-----

  ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py 543a31f61f4f711919b47689c1eb9b6c961df3ee 
  ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py 80965a60e23cbd9faaf43da6ca63ce5863ac5199 

Diff: https://reviews.apache.org/r/42452/diff/


Testing (updated)
-------

* Provision a 3 node cluster with HDFS, Hive, Ranger.
* Add Ranger after initial cluster bringup
* Shut down Ranger Admin
* Enable Ranger for HDFS
* Start Ranger Admin simultaneously with Namenode Restart

Observations :

* The retry logic is triggered
* Since Ranger is simultaneously coming up in another thread, we hit both the exceptions as shown in the log below.
* The component start is not blocked even if Ranger is not up

```
2016-01-18 07:18:40,583 - Rangeradmin: Skip ranger admin if it's down !
2016-01-18 07:18:40,587 - Will retry 4 time(s), caught exception: Connection to Ranger Admin failed. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s)
2016-01-18 07:19:08,597 - Will retry 3 time(s), caught exception: Connection to Ranger Admin failed. Reason - timeout. Sleeping for 8 sec(s)
2016-01-18 07:19:18,219 - amb_ranger_admin user is not present, creating user using given configurations
2016-01-18 07:19:18,621 - Ambari admin user creation successful.
2016-01-18 07:19:19,514 - Repository created Successfully
2016-01-18 07:19:20,194 - Policy updated Successfully
2016-01-18 07:19:20,195 - Ranger Repository created successfully and policies updated successfully providing ambari-qa user all permissions
```


Thanks,

Gautam Borad

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Andrew Onischuk <ao...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42452/#review115610
-----------------------------------------------------------



ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py (line 210)
<https://reviews.apache.org/r/42452/#comment176598>

    This will make ranger start more much longer, can we leave the sleep_time as is, but increase times count.


- Andrew Onischuk


On Jan. 21, 2016, 12:15 p.m., Gautam Borad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42452/
> -----------------------------------------------------------
> 
> (Updated Jan. 21, 2016, 12:15 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-14710
>     https://issues.apache.org/jira/browse/AMBARI-14710
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.
> 
> This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py c59dc8d 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py a72be33 
> 
> Diff: https://reviews.apache.org/r/42452/diff/
> 
> 
> Testing
> -------
> 
> * Provision a 3 node cluster with HDFS, Hive, Ranger.
> * Add Ranger after initial cluster bringup
> * Shut down Ranger Admin
> * Enable Ranger for HDFS
> * Start Ranger Admin simultaneously with Namenode Restart
> 
> Observations :
> 
> * The retry logic is triggered
> * Since Ranger is simultaneously coming up in another thread, we hit both the exceptions as shown in the log below.
> * The component start is not blocked even if Ranger is not up
> 
> ```
> 2016-01-18 07:18:40,583 - Rangeradmin: Skip ranger admin if it's down !
> 2016-01-18 07:18:40,587 - Will retry 4 time(s), caught exception: Connection to Ranger Admin failed. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s)
> 2016-01-18 07:19:08,597 - Will retry 3 time(s), caught exception: Connection to Ranger Admin failed. Reason - timeout. Sleeping for 8 sec(s)
> 2016-01-18 07:19:18,219 - amb_ranger_admin user is not present, creating user using given configurations
> 2016-01-18 07:19:18,621 - Ambari admin user creation successful.
> 2016-01-18 07:19:19,514 - Repository created Successfully
> 2016-01-18 07:19:20,194 - Policy updated Successfully
> 2016-01-18 07:19:20,195 - Ranger Repository created successfully and policies updated successfully providing ambari-qa user all permissions
> ```
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Sebastian Toader <st...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42452/#review115628
-----------------------------------------------------------

Ship it!


Ship It!

- Sebastian Toader


On Jan. 21, 2016, 2:46 p.m., Gautam Borad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42452/
> -----------------------------------------------------------
> 
> (Updated Jan. 21, 2016, 2:46 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-14710
>     https://issues.apache.org/jira/browse/AMBARI-14710
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.
> 
> This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py c59dc8d 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py a72be33 
> 
> Diff: https://reviews.apache.org/r/42452/diff/
> 
> 
> Testing
> -------
> 
> * Provision a 3 node cluster with HDFS, Hive, Ranger.
> * Add Ranger after initial cluster bringup
> * Shut down Ranger Admin
> * Enable Ranger for HDFS
> * Start Ranger Admin simultaneously with Namenode Restart
> 
> Observations :
> 
> * The retry logic is triggered
> * Since Ranger is simultaneously coming up in another thread, we hit both the exceptions as shown in the log below.
> * The component start is not blocked even if Ranger is not up
> 
> ```
> 2016-01-18 07:18:40,583 - Rangeradmin: Skip ranger admin if it's down !
> 2016-01-18 07:18:40,587 - Will retry 4 time(s), caught exception: Connection to Ranger Admin failed. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s)
> 2016-01-18 07:19:08,597 - Will retry 3 time(s), caught exception: Connection to Ranger Admin failed. Reason - timeout. Sleeping for 8 sec(s)
> 2016-01-18 07:19:18,219 - amb_ranger_admin user is not present, creating user using given configurations
> 2016-01-18 07:19:18,621 - Ambari admin user creation successful.
> 2016-01-18 07:19:19,514 - Repository created Successfully
> 2016-01-18 07:19:20,194 - Policy updated Successfully
> 2016-01-18 07:19:20,195 - Ranger Repository created successfully and policies updated successfully providing ambari-qa user all permissions
> ```
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Alejandro Fernandez <af...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42452/#review115655
-----------------------------------------------------------



ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py (line 210)
<https://reviews.apache.org/r/42452/#comment176650>

    The backoff_factor is 1.5, meaning that the sleeps will be for
    8
    12
    18
    ...
    307 (10th)
    
    Which even at 10 iterations sounds like over 10 mins.
    
    I suggest increasing the times to either 7 or 8, or changing the backoff_factor to 1 and times=10.


- Alejandro Fernandez


On Jan. 21, 2016, 1:46 p.m., Gautam Borad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42452/
> -----------------------------------------------------------
> 
> (Updated Jan. 21, 2016, 1:46 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-14710
>     https://issues.apache.org/jira/browse/AMBARI-14710
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.
> 
> This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py c59dc8d 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py a72be33 
> 
> Diff: https://reviews.apache.org/r/42452/diff/
> 
> 
> Testing
> -------
> 
> * Provision a 3 node cluster with HDFS, Hive, Ranger.
> * Add Ranger after initial cluster bringup
> * Shut down Ranger Admin
> * Enable Ranger for HDFS
> * Start Ranger Admin simultaneously with Namenode Restart
> 
> Observations :
> 
> * The retry logic is triggered
> * Since Ranger is simultaneously coming up in another thread, we hit both the exceptions as shown in the log below.
> * The component start is not blocked even if Ranger is not up
> 
> ```
> 2016-01-18 07:18:40,583 - Rangeradmin: Skip ranger admin if it's down !
> 2016-01-18 07:18:40,587 - Will retry 4 time(s), caught exception: Connection to Ranger Admin failed. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s)
> 2016-01-18 07:19:08,597 - Will retry 3 time(s), caught exception: Connection to Ranger Admin failed. Reason - timeout. Sleeping for 8 sec(s)
> 2016-01-18 07:19:18,219 - amb_ranger_admin user is not present, creating user using given configurations
> 2016-01-18 07:19:18,621 - Ambari admin user creation successful.
> 2016-01-18 07:19:19,514 - Repository created Successfully
> 2016-01-18 07:19:20,194 - Policy updated Successfully
> 2016-01-18 07:19:20,195 - Ranger Repository created successfully and policies updated successfully providing ambari-qa user all permissions
> ```
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Sebastian Toader <st...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42452/#review116077
-----------------------------------------------------------


Ship it!




Ship It!

- Sebastian Toader


On Jan. 25, 2016, 12:45 p.m., Gautam Borad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42452/
> -----------------------------------------------------------
> 
> (Updated Jan. 25, 2016, 12:45 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-14710
>     https://issues.apache.org/jira/browse/AMBARI-14710
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.
> 
> This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py e302235 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py 6624c89 
> 
> Diff: https://reviews.apache.org/r/42452/diff/
> 
> 
> Testing
> -------
> 
> * Provision a 3 node cluster with HDFS, Hive, Ranger.
> * Add Ranger after initial cluster bringup
> * Shut down Ranger Admin
> * Enable Ranger for HDFS
> * Start Ranger Admin simultaneously with Namenode Restart
> 
> Observations :
> 
> * The retry logic is triggered
> * Since Ranger is simultaneously coming up in another thread, we hit both the exceptions as shown in the log below.
> * The component start is not blocked even if Ranger is not up
> 
> ```
> 2016-01-18 07:18:40,583 - Rangeradmin: Skip ranger admin if it's down !
> 2016-01-18 07:18:40,587 - Will retry 4 time(s), caught exception: Connection to Ranger Admin failed. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s)
> 2016-01-18 07:19:08,597 - Will retry 3 time(s), caught exception: Connection to Ranger Admin failed. Reason - timeout. Sleeping for 8 sec(s)
> 2016-01-18 07:19:18,219 - amb_ranger_admin user is not present, creating user using given configurations
> 2016-01-18 07:19:18,621 - Ambari admin user creation successful.
> 2016-01-18 07:19:19,514 - Repository created Successfully
> 2016-01-18 07:19:20,194 - Policy updated Successfully
> 2016-01-18 07:19:20,195 - Ranger Repository created successfully and policies updated successfully providing ambari-qa user all permissions
> ```
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Andrew Onischuk <ao...@hortonworks.com>.


> On Jan. 25, 2016, 11:54 a.m., Andrew Onischuk wrote:
> > Ship It!

we just discussed offline, that marker solution will be implemented in a separate jira.


- Andrew


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42452/#review116075
-----------------------------------------------------------


On Jan. 25, 2016, 11:45 a.m., Gautam Borad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42452/
> -----------------------------------------------------------
> 
> (Updated Jan. 25, 2016, 11:45 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-14710
>     https://issues.apache.org/jira/browse/AMBARI-14710
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.
> 
> This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py e302235 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py 6624c89 
> 
> Diff: https://reviews.apache.org/r/42452/diff/
> 
> 
> Testing
> -------
> 
> * Provision a 3 node cluster with HDFS, Hive, Ranger.
> * Add Ranger after initial cluster bringup
> * Shut down Ranger Admin
> * Enable Ranger for HDFS
> * Start Ranger Admin simultaneously with Namenode Restart
> 
> Observations :
> 
> * The retry logic is triggered
> * Since Ranger is simultaneously coming up in another thread, we hit both the exceptions as shown in the log below.
> * The component start is not blocked even if Ranger is not up
> 
> ```
> 2016-01-18 07:18:40,583 - Rangeradmin: Skip ranger admin if it's down !
> 2016-01-18 07:18:40,587 - Will retry 4 time(s), caught exception: Connection to Ranger Admin failed. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s)
> 2016-01-18 07:19:08,597 - Will retry 3 time(s), caught exception: Connection to Ranger Admin failed. Reason - timeout. Sleeping for 8 sec(s)
> 2016-01-18 07:19:18,219 - amb_ranger_admin user is not present, creating user using given configurations
> 2016-01-18 07:19:18,621 - Ambari admin user creation successful.
> 2016-01-18 07:19:19,514 - Repository created Successfully
> 2016-01-18 07:19:20,194 - Policy updated Successfully
> 2016-01-18 07:19:20,195 - Ranger Repository created successfully and policies updated successfully providing ambari-qa user all permissions
> ```
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Andrew Onischuk <ao...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42452/#review116075
-----------------------------------------------------------


Ship it!




Ship It!

- Andrew Onischuk


On Jan. 25, 2016, 11:45 a.m., Gautam Borad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42452/
> -----------------------------------------------------------
> 
> (Updated Jan. 25, 2016, 11:45 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-14710
>     https://issues.apache.org/jira/browse/AMBARI-14710
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.
> 
> This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py e302235 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py 6624c89 
> 
> Diff: https://reviews.apache.org/r/42452/diff/
> 
> 
> Testing
> -------
> 
> * Provision a 3 node cluster with HDFS, Hive, Ranger.
> * Add Ranger after initial cluster bringup
> * Shut down Ranger Admin
> * Enable Ranger for HDFS
> * Start Ranger Admin simultaneously with Namenode Restart
> 
> Observations :
> 
> * The retry logic is triggered
> * Since Ranger is simultaneously coming up in another thread, we hit both the exceptions as shown in the log below.
> * The component start is not blocked even if Ranger is not up
> 
> ```
> 2016-01-18 07:18:40,583 - Rangeradmin: Skip ranger admin if it's down !
> 2016-01-18 07:18:40,587 - Will retry 4 time(s), caught exception: Connection to Ranger Admin failed. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s)
> 2016-01-18 07:19:08,597 - Will retry 3 time(s), caught exception: Connection to Ranger Admin failed. Reason - timeout. Sleeping for 8 sec(s)
> 2016-01-18 07:19:18,219 - amb_ranger_admin user is not present, creating user using given configurations
> 2016-01-18 07:19:18,621 - Ambari admin user creation successful.
> 2016-01-18 07:19:19,514 - Repository created Successfully
> 2016-01-18 07:19:20,194 - Policy updated Successfully
> 2016-01-18 07:19:20,195 - Ranger Repository created successfully and policies updated successfully providing ambari-qa user all permissions
> ```
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Andrew Onischuk <ao...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42452/#review116074
-----------------------------------------------------------



What about the marker file solution we talked about with Mehul? 
To prevent hanging of restart when ranger admin is stopped.

- Andrew Onischuk


On Jan. 25, 2016, 11:45 a.m., Gautam Borad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42452/
> -----------------------------------------------------------
> 
> (Updated Jan. 25, 2016, 11:45 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-14710
>     https://issues.apache.org/jira/browse/AMBARI-14710
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.
> 
> This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py e302235 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py 6624c89 
> 
> Diff: https://reviews.apache.org/r/42452/diff/
> 
> 
> Testing
> -------
> 
> * Provision a 3 node cluster with HDFS, Hive, Ranger.
> * Add Ranger after initial cluster bringup
> * Shut down Ranger Admin
> * Enable Ranger for HDFS
> * Start Ranger Admin simultaneously with Namenode Restart
> 
> Observations :
> 
> * The retry logic is triggered
> * Since Ranger is simultaneously coming up in another thread, we hit both the exceptions as shown in the log below.
> * The component start is not blocked even if Ranger is not up
> 
> ```
> 2016-01-18 07:18:40,583 - Rangeradmin: Skip ranger admin if it's down !
> 2016-01-18 07:18:40,587 - Will retry 4 time(s), caught exception: Connection to Ranger Admin failed. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s)
> 2016-01-18 07:19:08,597 - Will retry 3 time(s), caught exception: Connection to Ranger Admin failed. Reason - timeout. Sleeping for 8 sec(s)
> 2016-01-18 07:19:18,219 - amb_ranger_admin user is not present, creating user using given configurations
> 2016-01-18 07:19:18,621 - Ambari admin user creation successful.
> 2016-01-18 07:19:19,514 - Repository created Successfully
> 2016-01-18 07:19:20,194 - Policy updated Successfully
> 2016-01-18 07:19:20,195 - Ranger Repository created successfully and policies updated successfully providing ambari-qa user all permissions
> ```
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Gautam Borad <gb...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42452/
-----------------------------------------------------------

(Updated Jan. 25, 2016, 11:45 a.m.)

Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.

Changes
-------

Update the timeout to 10 mins and adjust the backoff_factor.

Bugs: AMBARI-14710
https://issues.apache.org/jira/browse/AMBARI-14710

Repository: ambari

Description
-------

Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.

This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.

Diffs (updated)
-----

ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py e302235
ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py 6624c89

Diff: https://reviews.apache.org/r/42452/diff/

Testing
-------

* Provision a 3 node cluster with HDFS, Hive, Ranger.
* Add Ranger after initial cluster bringup
* Shut down Ranger Admin
* Enable Ranger for HDFS
* Start Ranger Admin simultaneously with Namenode Restart

Observations :

* The retry logic is triggered
* Since Ranger is simultaneously coming up in another thread, we hit both the exceptions as shown in the log below.
* The component start is not blocked even if Ranger is not up

```
2016-01-18 07:18:40,583 - Rangeradmin: Skip ranger admin if it's down !
2016-01-18 07:18:40,587 - Will retry 4 time(s), caught exception: Connection to Ranger Admin failed. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s)
2016-01-18 07:19:08,597 - Will retry 3 time(s), caught exception: Connection to Ranger Admin failed. Reason - timeout. Sleeping for 8 sec(s)
2016-01-18 07:19:18,219 - amb_ranger_admin user is not present, creating user using given configurations
2016-01-18 07:19:18,621 - Ambari admin user creation successful.
2016-01-18 07:19:19,514 - Repository created Successfully
2016-01-18 07:19:20,194 - Policy updated Successfully
2016-01-18 07:19:20,195 - Ranger Repository created successfully and policies updated successfully providing ambari-qa user all permissions
```

Thanks,

Gautam Borad

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Andrew Onischuk <ao...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42452/#review115627
-----------------------------------------------------------

Ship it!


Ship It!

- Andrew Onischuk


On Jan. 21, 2016, 1:46 p.m., Gautam Borad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42452/
> -----------------------------------------------------------
> 
> (Updated Jan. 21, 2016, 1:46 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-14710
>     https://issues.apache.org/jira/browse/AMBARI-14710
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.
> 
> This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py c59dc8d 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py a72be33 
> 
> Diff: https://reviews.apache.org/r/42452/diff/
> 
> 
> Testing
> -------
> 
> * Provision a 3 node cluster with HDFS, Hive, Ranger.
> * Add Ranger after initial cluster bringup
> * Shut down Ranger Admin
> * Enable Ranger for HDFS
> * Start Ranger Admin simultaneously with Namenode Restart
> 
> Observations :
> 
> * The retry logic is triggered
> * Since Ranger is simultaneously coming up in another thread, we hit both the exceptions as shown in the log below.
> * The component start is not blocked even if Ranger is not up
> 
> ```
> 2016-01-18 07:18:40,583 - Rangeradmin: Skip ranger admin if it's down !
> 2016-01-18 07:18:40,587 - Will retry 4 time(s), caught exception: Connection to Ranger Admin failed. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s)
> 2016-01-18 07:19:08,597 - Will retry 3 time(s), caught exception: Connection to Ranger Admin failed. Reason - timeout. Sleeping for 8 sec(s)
> 2016-01-18 07:19:18,219 - amb_ranger_admin user is not present, creating user using given configurations
> 2016-01-18 07:19:18,621 - Ambari admin user creation successful.
> 2016-01-18 07:19:19,514 - Repository created Successfully
> 2016-01-18 07:19:20,194 - Policy updated Successfully
> 2016-01-18 07:19:20,195 - Ranger Repository created successfully and policies updated successfully providing ambari-qa user all permissions
> ```
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Velmurugan Periasamy <vp...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42452/#review115650
-----------------------------------------------------------

Ship it!


Ship It!

- Velmurugan Periasamy


On Jan. 21, 2016, 1:46 p.m., Gautam Borad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42452/
> -----------------------------------------------------------
> 
> (Updated Jan. 21, 2016, 1:46 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-14710
>     https://issues.apache.org/jira/browse/AMBARI-14710
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.
> 
> This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py c59dc8d 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py a72be33 
> 
> Diff: https://reviews.apache.org/r/42452/diff/
> 
> 
> Testing
> -------
> 
> * Provision a 3 node cluster with HDFS, Hive, Ranger.
> * Add Ranger after initial cluster bringup
> * Shut down Ranger Admin
> * Enable Ranger for HDFS
> * Start Ranger Admin simultaneously with Namenode Restart
> 
> Observations :
> 
> * The retry logic is triggered
> * Since Ranger is simultaneously coming up in another thread, we hit both the exceptions as shown in the log below.
> * The component start is not blocked even if Ranger is not up
> 
> ```
> 2016-01-18 07:18:40,583 - Rangeradmin: Skip ranger admin if it's down !
> 2016-01-18 07:18:40,587 - Will retry 4 time(s), caught exception: Connection to Ranger Admin failed. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s)
> 2016-01-18 07:19:08,597 - Will retry 3 time(s), caught exception: Connection to Ranger Admin failed. Reason - timeout. Sleeping for 8 sec(s)
> 2016-01-18 07:19:18,219 - amb_ranger_admin user is not present, creating user using given configurations
> 2016-01-18 07:19:18,621 - Ambari admin user creation successful.
> 2016-01-18 07:19:19,514 - Repository created Successfully
> 2016-01-18 07:19:20,194 - Policy updated Successfully
> 2016-01-18 07:19:20,195 - Ranger Repository created successfully and policies updated successfully providing ambari-qa user all permissions
> ```
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Gautam Borad <gb...@gmail.com>.

(Updated Jan. 21, 2016, 1:46 p.m.)

Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.

Changes
-------

As suggested by Andrew Onishuk, increasing number of Re-tries rather than increasing sleep_time.

Bugs: AMBARI-14710
https://issues.apache.org/jira/browse/AMBARI-14710

Repository: ambari

Description
-------

Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.

This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.

Diffs (updated)
-----

ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py c59dc8d
ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py a72be33

Diff: https://reviews.apache.org/r/42452/diff/

Testing
-------

Observations :

Thanks,

Gautam Borad

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Gautam Borad <gb...@gmail.com>.

(Updated Jan. 21, 2016, 12:15 p.m.)

Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.

Changes
-------

Adding increased sleep_time=30 for retries for REST calls, to take care of simultaneus restarts.

Bugs: AMBARI-14710
https://issues.apache.org/jira/browse/AMBARI-14710

Repository: ambari

Description
-------

Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.

This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.

Diffs (updated)
-----

ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py c59dc8d
ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py a72be33

Diff: https://reviews.apache.org/r/42452/diff/

Testing
-------

Observations :

Thanks,

Gautam Borad

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Sebastian Toader <st...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42452/#review115022
-----------------------------------------------------------

Ship it!


Ship It!

- Sebastian Toader


On Jan. 18, 2016, 6 p.m., Gautam Borad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42452/
> -----------------------------------------------------------
> 
> (Updated Jan. 18, 2016, 6 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-14710
>     https://issues.apache.org/jira/browse/AMBARI-14710
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.
> 
> This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py 543a31f61f4f711919b47689c1eb9b6c961df3ee 
>   ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py 80965a60e23cbd9faaf43da6ca63ce5863ac5199 
> 
> Diff: https://reviews.apache.org/r/42452/diff/
> 
> 
> Testing
> -------
> 
> * Provision a 3 node cluster with HDFS, Hive, Ranger.
> * Add Ranger after initial cluster bringup
> * Shut down Ranger Admin
> * Enable Ranger for HDFS
> * Start Ranger Admin simultaneously with Namenode Restart
> 
> Observations :
> 
> * The retry logic is triggered
> * Since Ranger is simultaneously coming up in another thread, we hit both the exceptions as shown in the log below.
> * The component start is not blocked even if Ranger is not up
> 
> ```
> 2016-01-18 07:18:40,583 - Rangeradmin: Skip ranger admin if it's down !
> 2016-01-18 07:18:40,587 - Will retry 4 time(s), caught exception: Connection to Ranger Admin failed. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s)
> 2016-01-18 07:19:08,597 - Will retry 3 time(s), caught exception: Connection to Ranger Admin failed. Reason - timeout. Sleeping for 8 sec(s)
> 2016-01-18 07:19:18,219 - amb_ranger_admin user is not present, creating user using given configurations
> 2016-01-18 07:19:18,621 - Ambari admin user creation successful.
> 2016-01-18 07:19:19,514 - Repository created Successfully
> 2016-01-18 07:19:20,194 - Policy updated Successfully
> 2016-01-18 07:19:20,195 - Ranger Repository created successfully and policies updated successfully providing ambari-qa user all permissions
> ```
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>

Re: Review Request 42452: AMBARI-14710 : Add retry logic for REST calls between Ambari and Ranger

Posted by Gautam Borad <gb...@gmail.com>.

(Updated Jan. 18, 2016, 5 p.m.)

Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, Sebastian Toader, and Velmurugan Periasamy.

Changes
-------

Fix the Bugs field with right AMBARI jira.

Bugs: AMBARI-14710
https://issues.apache.org/jira/browse/AMBARI-14710

Repository: ambari

Description
-------

Current implementation of check_ranger_login_urllib2 in ranger_functions.py and ranger_functions_v2.py triggers the retry logic under the safe_retry decorator only for the Timeout error.

This logic should be triggered for the exceptions also, to ensure that Ranger Admin is given enough time to start up before giving up.

Diffs
-----

ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py 543a31f61f4f711919b47689c1eb9b6c961df3ee
ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py 80965a60e23cbd9faaf43da6ca63ce5863ac5199

Diff: https://reviews.apache.org/r/42452/diff/

Testing
-------

Observations :

Thanks,

Gautam Borad