You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Alok Lal <al...@hortonworks.com> on 2015/08/18 08:42:26 UTC
Review Request 37565: Audit to db: Truncate all string values of audit
record so that writing of audit does not fail
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37565/
-----------------------------------------------------------
Review request for ranger and Madhan Neethiraj.
Bugs: RANGER-615
https://issues.apache.org/jira/browse/RANGER-615
Repository: ranger
Description
-------
Audit to db: if audit information exceeds column size then insertion of audit record fails. Longer values should be truncated and allow to easily control the truncated size
- All string columns should be truncated. Once truncated the value should have some marker that’ll show that it was truncated.
- Length at which truncation occurs should be configurable perhaps by db-audit-provider sections of audit properties file.
- While at it we could also add the ability to suppress truncation or prevent a column from getting into audit all together.
- All of this should be contained to audit to db, e.g. audits written to HDFS or Solr should never get truncated.
This change is done on HEAD of 0.5. I'll port it later to master.
Diffs
-----
agents-audit/src/main/java/org/apache/ranger/audit/destination/DBAuditDestination.java 3d31c06
agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java 435393e
agents-audit/src/main/java/org/apache/ranger/audit/model/AuditEventBase.java 2c6a87f
agents-audit/src/main/java/org/apache/ranger/audit/model/AuthzAuditEvent.java d648de3
agents-audit/src/main/java/org/apache/ranger/audit/provider/DbAuditProvider.java d475f89
Diff: https://reviews.apache.org/r/37565/diff/
Testing
-------
Reduced column size of resource_path field down to a smaller value and reproduced the problem with HDFS. Then validated that:
- Setting max column size property in audit truncates the resource_path. It has truncation marker in it.
- Setting the column size property to 0 prevents anything from getting output to the column.
- Setting the column size to -1 bring the problem back since truncation gets suppressed.
Thanks,
Alok Lal
Re: Review Request 37565: Audit to db: Truncate all string values of
audit record so that writing of audit does not fail
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37565/#review96024
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On Aug. 21, 2015, 2:11 a.m., Alok Lal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37565/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2015, 2:11 a.m.)
>
>
> Review request for ranger and Madhan Neethiraj.
>
>
> Bugs: RANGER-615
> https://issues.apache.org/jira/browse/RANGER-615
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Audit to db: if audit information exceeds column size then insertion of audit record fails. Longer values should be truncated and allow to easily control the truncated size
>
> - All string columns should be truncated. Once truncated the value should have some marker that’ll show that it was truncated.
> - Length at which truncation occurs should be configurable perhaps by db-audit-provider sections of audit properties file.
> - While at it we could also add the ability to suppress truncation or prevent a column from getting into audit all together.
> - All of this should be contained to audit to db, e.g. audits written to HDFS or Solr should never get truncated.
>
> This change is done on HEAD of 0.5. I'll port it later to master.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java 435393e
>
> Diff: https://reviews.apache.org/r/37565/diff/
>
>
> Testing
> -------
>
> Reduced column size of resource_path field down to a smaller value and reproduced the problem with HDFS. Then validated that:
> - Setting max column size property in audit truncates the resource_path. It has truncation marker in it.
> - Setting the column size property to 0 prevents anything from getting output to the column.
> - Setting the column size to -1 bring the problem back since truncation gets suppressed.
>
>
> Thanks,
>
> Alok Lal
>
>
Re: Review Request 37565: Audit to db: Truncate all string values of
audit record so that writing of audit does not fail
Posted by Alok Lal <al...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37565/
-----------------------------------------------------------
(Updated Aug. 20, 2015, 7:11 p.m.)
Review request for ranger and Madhan Neethiraj.
Changes
-------
updated the default length of Requestdata and requestpath columns that were increased in 0.5 from 2k to 4k.
Bugs: RANGER-615
https://issues.apache.org/jira/browse/RANGER-615
Repository: ranger
Description
-------
Audit to db: if audit information exceeds column size then insertion of audit record fails. Longer values should be truncated and allow to easily control the truncated size
- All string columns should be truncated. Once truncated the value should have some marker that’ll show that it was truncated.
- Length at which truncation occurs should be configurable perhaps by db-audit-provider sections of audit properties file.
- While at it we could also add the ability to suppress truncation or prevent a column from getting into audit all together.
- All of this should be contained to audit to db, e.g. audits written to HDFS or Solr should never get truncated.
This change is done on HEAD of 0.5. I'll port it later to master.
Diffs (updated)
-----
agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java 435393e
Diff: https://reviews.apache.org/r/37565/diff/
Testing
-------
Reduced column size of resource_path field down to a smaller value and reproduced the problem with HDFS. Then validated that:
- Setting max column size property in audit truncates the resource_path. It has truncation marker in it.
- Setting the column size property to 0 prevents anything from getting output to the column.
- Setting the column size to -1 bring the problem back since truncation gets suppressed.
Thanks,
Alok Lal
Re: Review Request 37565: Audit to db: Truncate all string values of
audit record so that writing of audit does not fail
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37565/#review96021
-----------------------------------------------------------
agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java (line 64)
<https://reviews.apache.org/r/37565/#comment151216>
The column size for resource_path and request_data is set to 4000 in ranger-0.5. See DB patch script: security-admin/db/mysql/patches/audit/011-auditcolumnssize.sql. Please review.
- Madhan Neethiraj
On Aug. 20, 2015, 2:17 a.m., Alok Lal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37565/
> -----------------------------------------------------------
>
> (Updated Aug. 20, 2015, 2:17 a.m.)
>
>
> Review request for ranger and Madhan Neethiraj.
>
>
> Bugs: RANGER-615
> https://issues.apache.org/jira/browse/RANGER-615
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Audit to db: if audit information exceeds column size then insertion of audit record fails. Longer values should be truncated and allow to easily control the truncated size
>
> - All string columns should be truncated. Once truncated the value should have some marker that’ll show that it was truncated.
> - Length at which truncation occurs should be configurable perhaps by db-audit-provider sections of audit properties file.
> - While at it we could also add the ability to suppress truncation or prevent a column from getting into audit all together.
> - All of this should be contained to audit to db, e.g. audits written to HDFS or Solr should never get truncated.
>
> This change is done on HEAD of 0.5. I'll port it later to master.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java 435393e
>
> Diff: https://reviews.apache.org/r/37565/diff/
>
>
> Testing
> -------
>
> Reduced column size of resource_path field down to a smaller value and reproduced the problem with HDFS. Then validated that:
> - Setting max column size property in audit truncates the resource_path. It has truncation marker in it.
> - Setting the column size property to 0 prevents anything from getting output to the column.
> - Setting the column size to -1 bring the problem back since truncation gets suppressed.
>
>
> Thanks,
>
> Alok Lal
>
>
Re: Review Request 37565: Audit to db: Truncate all string values of
audit record so that writing of audit does not fail
Posted by Alok Lal <al...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37565/
-----------------------------------------------------------
(Updated Aug. 19, 2015, 7:17 p.m.)
Review request for ranger and Madhan Neethiraj.
Changes
-------
Changed the variable names in 0.5 to line up with v3 audit naming convention, e.g. xasecure.audit.destination.db.max.column.length.resource_path. Repeated the same sets of tests to validate the change.
Bugs: RANGER-615
https://issues.apache.org/jira/browse/RANGER-615
Repository: ranger
Description
-------
Audit to db: if audit information exceeds column size then insertion of audit record fails. Longer values should be truncated and allow to easily control the truncated size
- All string columns should be truncated. Once truncated the value should have some marker that’ll show that it was truncated.
- Length at which truncation occurs should be configurable perhaps by db-audit-provider sections of audit properties file.
- While at it we could also add the ability to suppress truncation or prevent a column from getting into audit all together.
- All of this should be contained to audit to db, e.g. audits written to HDFS or Solr should never get truncated.
This change is done on HEAD of 0.5. I'll port it later to master.
Diffs (updated)
-----
agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java 435393e
Diff: https://reviews.apache.org/r/37565/diff/
Testing
-------
Reduced column size of resource_path field down to a smaller value and reproduced the problem with HDFS. Then validated that:
- Setting max column size property in audit truncates the resource_path. It has truncation marker in it.
- Setting the column size property to 0 prevents anything from getting output to the column.
- Setting the column size to -1 bring the problem back since truncation gets suppressed.
Thanks,
Alok Lal