You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Mahesh Bandal <ma...@gmail.com> on 2021/11/19 12:48:36 UTC
Review Request 73717: RANGER-3518: Limit the query size stored in Audit logs
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73717/
-----------------------------------------------------------
Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-3518
https://issues.apache.org/jira/browse/RANGER-3518
Repository: ranger
Description
-------
Ranger audit stores complete query as it is in audit logs. If the single query size is too large (i.e reqData > 10MB) it consumes large heap size on hive plugin size. When similar 200-300 queries are executed simultaneously, ranger audit takes up more than half of the heap.
With this patch we can configure size of the request data stored in audit logs.
We can set value for config "xasecure.audit.solr.limit.query.req.size" in ranger-hive-audit.xml
Recommended value is 1024; however default value is set to Integer.MAX_VALUE
Diffs
-----
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java a3d575c86
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java 2be442460
Diff: https://reviews.apache.org/r/73717/diff/1/
Testing
-------
BUILD SUCCESS: mvn clean compile verify test install
Ranger installation and setup successful.
Generated large hive queries and validated that audits in spool file stores query size equal to the value mentioned in config "xasecure.audit.solr.limit.query.req.size".
Thanks,
Mahesh Bandal
Re: Review Request 73717: RANGER-3518: Limit the query size stored in Audit logs
Posted by Mahesh Bandal <ma...@gmail.com>.
> On Nov. 25, 2021, 3:07 a.m., Pradeep Agrawal wrote:
> > hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java
> > Lines 45 (patched)
> > <https://reviews.apache.org/r/73717/diff/1/?file=2255874#file2255874line45>
> >
> > Can you improve this to work for elasticsearch as well.
We can track changes for elasticsearch separately in a new JIRA ticket.
- Mahesh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73717/#review223770
-----------------------------------------------------------
On Nov. 19, 2021, 1:53 p.m., Mahesh Bandal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73717/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2021, 1:53 p.m.)
>
>
> Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3518
> https://issues.apache.org/jira/browse/RANGER-3518
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Ranger audit stores complete query as it is in audit logs. If the single query size is too large (i.e reqData > 10MB) it consumes large heap size on hive plugin size. When similar 200-300 queries are executed simultaneously, ranger audit takes up more than half of the heap.
>
> With this patch we can configure size of the request data stored in audit logs.
> We can set value for config "xasecure.audit.solr.limit.query.req.size" in ranger-hive-audit.xml
> Recommended value is 1024; however default value is set to Integer.MAX_VALUE
>
>
> Diffs
> -----
>
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java a3d575c86
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java 2be442460
>
>
> Diff: https://reviews.apache.org/r/73717/diff/1/
>
>
> Testing
> -------
>
> BUILD SUCCESS: mvn clean compile verify test install
> Ranger installation and setup successful.
> Generated large hive queries and validated that audits in spool file stores query size equal to the value mentioned in config "xasecure.audit.solr.limit.query.req.size"
>
>
> Thanks,
>
> Mahesh Bandal
>
>
Re: Review Request 73717: RANGER-3518: Limit the query size stored in Audit logs
Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73717/#review223770
-----------------------------------------------------------
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java
Lines 45 (patched)
<https://reviews.apache.org/r/73717/#comment312853>
Can you improve this to work for elasticsearch as well.
- Pradeep Agrawal
On Nov. 19, 2021, 1:53 p.m., Mahesh Bandal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73717/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2021, 1:53 p.m.)
>
>
> Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3518
> https://issues.apache.org/jira/browse/RANGER-3518
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Ranger audit stores complete query as it is in audit logs. If the single query size is too large (i.e reqData > 10MB) it consumes large heap size on hive plugin size. When similar 200-300 queries are executed simultaneously, ranger audit takes up more than half of the heap.
>
> With this patch we can configure size of the request data stored in audit logs.
> We can set value for config "xasecure.audit.solr.limit.query.req.size" in ranger-hive-audit.xml
> Recommended value is 1024; however default value is set to Integer.MAX_VALUE
>
>
> Diffs
> -----
>
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java a3d575c86
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java 2be442460
>
>
> Diff: https://reviews.apache.org/r/73717/diff/1/
>
>
> Testing
> -------
>
> BUILD SUCCESS: mvn clean compile verify test install
> Ranger installation and setup successful.
> Generated large hive queries and validated that audits in spool file stores query size equal to the value mentioned in config "xasecure.audit.solr.limit.query.req.size"
>
>
> Thanks,
>
> Mahesh Bandal
>
>
Re: Review Request 73717: RANGER-3518: Limit the query size stored in Audit logs
Posted by Ramesh Mani <rm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73717/#review223768
-----------------------------------------------------------
Ship it!
Ship It!
- Ramesh Mani
On Nov. 19, 2021, 1:53 p.m., Mahesh Bandal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73717/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2021, 1:53 p.m.)
>
>
> Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3518
> https://issues.apache.org/jira/browse/RANGER-3518
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Ranger audit stores complete query as it is in audit logs. If the single query size is too large (i.e reqData > 10MB) it consumes large heap size on hive plugin size. When similar 200-300 queries are executed simultaneously, ranger audit takes up more than half of the heap.
>
> With this patch we can configure size of the request data stored in audit logs.
> We can set value for config "xasecure.audit.solr.limit.query.req.size" in ranger-hive-audit.xml
> Recommended value is 1024; however default value is set to Integer.MAX_VALUE
>
>
> Diffs
> -----
>
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java a3d575c86
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java 2be442460
>
>
> Diff: https://reviews.apache.org/r/73717/diff/1/
>
>
> Testing
> -------
>
> BUILD SUCCESS: mvn clean compile verify test install
> Ranger installation and setup successful.
> Generated large hive queries and validated that audits in spool file stores query size equal to the value mentioned in config "xasecure.audit.solr.limit.query.req.size"
>
>
> Thanks,
>
> Mahesh Bandal
>
>
Re: Review Request 73717: RANGER-3518: Limit the query size stored in Audit logs
Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73717/#review223782
-----------------------------------------------------------
Ship it!
Ship It!
- Pradeep Agrawal
On Nov. 30, 2021, 1:32 p.m., Mahesh Bandal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73717/
> -----------------------------------------------------------
>
> (Updated Nov. 30, 2021, 1:32 p.m.)
>
>
> Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3518
> https://issues.apache.org/jira/browse/RANGER-3518
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Ranger audit stores complete query as it is in audit logs. If the single query size is too large (i.e reqData > 10MB) it consumes large heap size on hive plugin size. When similar 200-300 queries are executed simultaneously, ranger audit takes up more than half of the heap.
>
> With this patch we can configure size of the request data stored in audit logs.
> We can set value for config "xasecure.audit.solr.limit.query.req.size" in ranger-hive-audit.xml
> Recommended value is 1024; however default value is set to Integer.MAX_VALUE
>
>
> Diffs
> -----
>
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java 5c04bdb69
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java 2be442460
>
>
> Diff: https://reviews.apache.org/r/73717/diff/2/
>
>
> Testing
> -------
>
> BUILD SUCCESS: mvn clean compile verify test install
> Ranger installation and setup successful.
> Generated large hive queries and validated that audits in spool file stores query size equal to the value mentioned in config "xasecure.audit.solr.limit.query.req.size"
>
>
> Thanks,
>
> Mahesh Bandal
>
>
Re: Review Request 73717: RANGER-3518: Limit the query size stored in Audit logs
Posted by Mahesh Bandal <ma...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73717/
-----------------------------------------------------------
(Updated Nov. 30, 2021, 1:32 p.m.)
Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-3518
https://issues.apache.org/jira/browse/RANGER-3518
Repository: ranger
Description
-------
Ranger audit stores complete query as it is in audit logs. If the single query size is too large (i.e reqData > 10MB) it consumes large heap size on hive plugin size. When similar 200-300 queries are executed simultaneously, ranger audit takes up more than half of the heap.
With this patch we can configure size of the request data stored in audit logs.
We can set value for config "xasecure.audit.solr.limit.query.req.size" in ranger-hive-audit.xml
Recommended value is 1024; however default value is set to Integer.MAX_VALUE
Diffs (updated)
-----
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java 5c04bdb69
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java 2be442460
Diff: https://reviews.apache.org/r/73717/diff/2/
Changes: https://reviews.apache.org/r/73717/diff/1-2/
Testing
-------
BUILD SUCCESS: mvn clean compile verify test install
Ranger installation and setup successful.
Generated large hive queries and validated that audits in spool file stores query size equal to the value mentioned in config "xasecure.audit.solr.limit.query.req.size"
Thanks,
Mahesh Bandal
Re: Review Request 73717: RANGER-3518: Limit the query size stored in Audit logs
Posted by Mahesh Bandal <ma...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73717/
-----------------------------------------------------------
(Updated Nov. 19, 2021, 1:53 p.m.)
Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-3518
https://issues.apache.org/jira/browse/RANGER-3518
Repository: ranger
Description
-------
Ranger audit stores complete query as it is in audit logs. If the single query size is too large (i.e reqData > 10MB) it consumes large heap size on hive plugin size. When similar 200-300 queries are executed simultaneously, ranger audit takes up more than half of the heap.
With this patch we can configure size of the request data stored in audit logs.
We can set value for config "xasecure.audit.solr.limit.query.req.size" in ranger-hive-audit.xml
Recommended value is 1024; however default value is set to Integer.MAX_VALUE
Diffs
-----
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java a3d575c86
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java 2be442460
Diff: https://reviews.apache.org/r/73717/diff/1/
Testing (updated)
-------
BUILD SUCCESS: mvn clean compile verify test install
Ranger installation and setup successful.
Generated large hive queries and validated that audits in spool file stores query size equal to the value mentioned in config "xasecure.audit.solr.limit.query.req.size"
Thanks,
Mahesh Bandal