You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ranger.apache.org by Pradeep Agrawal <pr...@freestoneinfotech.com> on 2017/03/10 06:17:28 UTC

Review Request 57496: RANGER-1440 : Improve install script to retry failing statement

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57496/
-----------------------------------------------------------

Review request for ranger, Ankita Sinha, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.

Bugs: RANGER-1440
https://issues.apache.org/jira/browse/RANGER-1440

Repository: ranger

Description
-------

**Problem Statement:**
Ranger installation may fail due to network latency or during parallel install in Ranger HA environment.
**Proposed Solution:**
Ranger installation script can be improved to retry failed sql statements(specifically create/update request).
Rather calling subprocess.call() only once for all such statements execution wrote a function subprocessCallWithRetry() which can accept statement and retry 2 more times if fails in first attempt.

**Other changes :**
Patch wait time is changed to 2 minute from 5 minute.

**Note:** Ranger installation script failure is still possible due to poor network latency of db server.
For example: if install process of two different node read same data at particular timestamp but while updating the same data one node could update that data quickly due to better network latency and if at the same interval second machine got poor latency then second node process might try to update the old data which have already been changed by first node.

Diffs
-----

security-admin/scripts/db_setup.py 6d1a047
security-admin/scripts/dba_script.py 1f99328
security-admin/scripts/setup.sh 76bb119
security-admin/scripts/update_property.py 338fbf5

Diff: https://reviews.apache.org/r/57496/diff/1/

Testing
-------

**Steps performed(with patch):** Below steps have been tested for all db flavor.
1. Created Build with patch and untar the build on two node.
2. Opened install.properties and provided same db configuration in install.properties of both ranger-admin nodes.
3. Called setup.sh file parallelly from both hosts.

**Expected behavior :** If same operation or sql statement is executed from both node then either of the node script may break and retry the script or sleep for some time to reduce the possibility of ranger install failure. Ranger admin ui should get loaded on browser whenever Ranger url is requested.

**Expected behavior :** Ranger install worked fine. it was observed that few create/update sql statements failed in middle of the script execution; but the installation resumed and finished well. Was able to view and login to Ranger admin.

Tried above steps 5 times for each db flavours.

Thanks,

Pradeep Agrawal

Re: Review Request 57496: RANGER-1440 : Improve install script to retry failing statement

Posted by Velmurugan Periasamy <vp...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57496/#review168635
-----------------------------------------------------------


Ship it!




Ship It!

- Velmurugan Periasamy


On March 10, 2017, 6:21 a.m., Pradeep Agrawal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57496/
> -----------------------------------------------------------
> 
> (Updated March 10, 2017, 6:21 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-1440
>     https://issues.apache.org/jira/browse/RANGER-1440
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> **Problem Statement:** 
> Ranger installation may fail due to network latency or during parallel install in Ranger HA environment.
> **Proposed Solution:** 
> Ranger installation script can be improved to retry failed sql statements(specifically create/update request).
> Rather calling subprocess.call() only once for all such statements executions, wrote a function subprocessCallWithRetry() which can accept statement and retry 2 more times if fails in first attempt.
> 
> **Other changes :**
> Patch wait time is changed to 2 minute from 5 minute.
> 
> **Note:** Ranger installation script failure is still possible due to poor network latency of db server. 
> For example: if install process of two different node read same data at particular timestamp but while updating the same data one node could update that data quickly due to better network latency and if at the same interval second machine got poor latency then second node process might try to update the old data which have already been changed by first node.
> 
> 
> Diffs
> -----
> 
>   security-admin/scripts/db_setup.py 6d1a047 
>   security-admin/scripts/dba_script.py 1f99328 
>   security-admin/scripts/setup.sh 76bb119 
>   security-admin/scripts/update_property.py 338fbf5 
> 
> 
> Diff: https://reviews.apache.org/r/57496/diff/1/
> 
> 
> Testing
> -------
> 
> **Steps performed(with patch):** Below steps have been tested for all db flavor.
> 1. Created Build with patch and untar the build on two node.
> 2. Opened install.properties and provided same db configuration in install.properties of both ranger-admin nodes.
> 3. Called setup.sh file parallelly from both hosts.
> 
> **Expected behavior :**  If same operation or sql statement is executed from both node then either of the node script may break and retry the script or sleep for some time to reduce the possibility of ranger install failure. Ranger admin ui should get loaded on browser whenever Ranger url is requested.
> 
> **Actual behavior :**  Ranger install worked fine. it was observed that few create/update sql statements failed in middle of the script execution; but the installation resumed and finished well. Was able to view and login to Ranger admin.
> 
> Tried above steps 5 times for each db flavours.
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>

Re: Review Request 57496: RANGER-1440 : Improve install script to retry failing statement

Posted by Gautam Borad <gb...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57496/#review168636
-----------------------------------------------------------


Ship it!




Ship It!

- Gautam Borad


On March 10, 2017, 6:21 a.m., Pradeep Agrawal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57496/
> -----------------------------------------------------------
> 
> (Updated March 10, 2017, 6:21 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-1440
>     https://issues.apache.org/jira/browse/RANGER-1440
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> **Problem Statement:** 
> Ranger installation may fail due to network latency or during parallel install in Ranger HA environment.
> **Proposed Solution:** 
> Ranger installation script can be improved to retry failed sql statements(specifically create/update request).
> Rather calling subprocess.call() only once for all such statements executions, wrote a function subprocessCallWithRetry() which can accept statement and retry 2 more times if fails in first attempt.
> 
> **Other changes :**
> Patch wait time is changed to 2 minute from 5 minute.
> 
> **Note:** Ranger installation script failure is still possible due to poor network latency of db server. 
> For example: if install process of two different node read same data at particular timestamp but while updating the same data one node could update that data quickly due to better network latency and if at the same interval second machine got poor latency then second node process might try to update the old data which have already been changed by first node.
> 
> 
> Diffs
> -----
> 
>   security-admin/scripts/db_setup.py 6d1a047 
>   security-admin/scripts/dba_script.py 1f99328 
>   security-admin/scripts/setup.sh 76bb119 
>   security-admin/scripts/update_property.py 338fbf5 
> 
> 
> Diff: https://reviews.apache.org/r/57496/diff/1/
> 
> 
> Testing
> -------
> 
> **Steps performed(with patch):** Below steps have been tested for all db flavor.
> 1. Created Build with patch and untar the build on two node.
> 2. Opened install.properties and provided same db configuration in install.properties of both ranger-admin nodes.
> 3. Called setup.sh file parallelly from both hosts.
> 
> **Expected behavior :**  If same operation or sql statement is executed from both node then either of the node script may break and retry the script or sleep for some time to reduce the possibility of ranger install failure. Ranger admin ui should get loaded on browser whenever Ranger url is requested.
> 
> **Actual behavior :**  Ranger install worked fine. it was observed that few create/update sql statements failed in middle of the script execution; but the installation resumed and finished well. Was able to view and login to Ranger admin.
> 
> Tried above steps 5 times for each db flavours.
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>

Re: Review Request 57496: RANGER-1440 : Improve install script to retry failing statement

Posted by Pradeep Agrawal <pr...@freestoneinfotech.com>.

(Updated March 10, 2017, 6:21 a.m.)

Bugs: RANGER-1440
https://issues.apache.org/jira/browse/RANGER-1440

Repository: ranger

Description (updated)
-------

**Problem Statement:**
Ranger installation may fail due to network latency or during parallel install in Ranger HA environment.
**Proposed Solution:**
Ranger installation script can be improved to retry failed sql statements(specifically create/update request).
Rather calling subprocess.call() only once for all such statements executions, wrote a function subprocessCallWithRetry() which can accept statement and retry 2 more times if fails in first attempt.

**Other changes :**
Patch wait time is changed to 2 minute from 5 minute.

Diffs
-----

security-admin/scripts/db_setup.py 6d1a047
security-admin/scripts/dba_script.py 1f99328
security-admin/scripts/setup.sh 76bb119
security-admin/scripts/update_property.py 338fbf5

Diff: https://reviews.apache.org/r/57496/diff/1/

Testing (updated)
-------

**Actual behavior :** Ranger install worked fine. it was observed that few create/update sql statements failed in middle of the script execution; but the installation resumed and finished well. Was able to view and login to Ranger admin.

Tried above steps 5 times for each db flavours.

Thanks,

Pradeep Agrawal