You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Sarath Subramanian <sa...@gmail.com> on 2017/01/16 22:21:24 UTC
Review Request 55592: [ATLAS-1463] Mechanism to exclude specified
entity
attributes from getting stored in audit repository during entity
create/update/delete operations
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55592/
-----------------------------------------------------------
Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Suma Shivaprasad.
Bugs: ATLAS-1463
https://issues.apache.org/jira/browse/ATLAS-1463
Repository: atlas
Description
-------
Atlas uses HBase as its store for audit repository and when new entities are created the entity definition string along with all its attributes are stored in audit repository. When size of the entity is greater than "hbase.client.keyvalue.maxsize" property then entity creation fails with "KeyValue size too large" exception.
A property needs to be set in atlas-application.properties to specify list of attributes to be excluded from getting stored in audit repo. Also check needs to be made to make sure entity size is not greater than keyvalue max size.
This patch excludes specified attribute from getting stored in audit repository and also does a size check for max allowed entity size to be stored in the audit repo.
Diffs
-----
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java 958ecaf
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java 417092a
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 5099521
repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java cf76596
repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java 9f77bfe
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 9a9beff
Diff: https://reviews.apache.org/r/55592/diff/
Testing
-------
Tested using POSTMAN REST Client. Created large entities and made sure entities created fine.
Thanks,
Sarath Subramanian
Re: Review Request 55592: [ATLAS-1463] Mechanism to exclude specified
entity
attributes from getting stored in audit repository during entity
create/update/delete operations
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55592/#review162355
-----------------------------------------------------------
Fix it, then Ship it!
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java (line 133)
<https://reviews.apache.org/r/55592/#comment233709>
Given that -1 is the value returned from InMemoryEntityAuditRepository.java and NoopEntityAuditRepository.java, consider changing this condition to:
if (auditMaxSize > 0 && auditSize > auditMaxSize)
- Madhan Neethiraj
On Jan. 19, 2017, 9:51 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55592/
> -----------------------------------------------------------
>
> (Updated Jan. 19, 2017, 9:51 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Suma Shivaprasad.
>
>
> Bugs: ATLAS-1463
> https://issues.apache.org/jira/browse/ATLAS-1463
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas uses HBase as its store for audit repository and when new entities are created the entity definition string along with all its attributes are stored in audit repository. When size of the entity is greater than "hbase.client.keyvalue.maxsize" property then entity creation fails with "KeyValue size too large" exception.
> A property needs to be set in atlas-application.properties to specify list of attributes to be excluded from getting stored in audit repo. Also check needs to be made to make sure entity size is not greater than keyvalue max size.
>
> This patch excludes specified attribute from getting stored in audit repository and also does a size check for max allowed entity size to be stored in the audit repo.
>
>
> Diffs
> -----
>
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java 958ecaf
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java 417092a
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 5099521
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java cf76596
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java 9f77bfe
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 9a9beff
>
> Diff: https://reviews.apache.org/r/55592/diff/
>
>
> Testing
> -------
>
> Tested using POSTMAN REST Client. Created large entities and made sure entities created fine.
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 55592: [ATLAS-1463] Mechanism to exclude specified
entity
attributes from getting stored in audit repository during entity
create/update/delete operations
Posted by Sarath Subramanian <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55592/
-----------------------------------------------------------
(Updated Jan. 19, 2017, 1:51 p.m.)
Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Suma Shivaprasad.
Changes
-------
1. Addressed review comments
2. Abstracted HBase specific imlementations to its own class
Bugs: ATLAS-1463
https://issues.apache.org/jira/browse/ATLAS-1463
Repository: atlas
Description
-------
Atlas uses HBase as its store for audit repository and when new entities are created the entity definition string along with all its attributes are stored in audit repository. When size of the entity is greater than "hbase.client.keyvalue.maxsize" property then entity creation fails with "KeyValue size too large" exception.
A property needs to be set in atlas-application.properties to specify list of attributes to be excluded from getting stored in audit repo. Also check needs to be made to make sure entity size is not greater than keyvalue max size.
This patch excludes specified attribute from getting stored in audit repository and also does a size check for max allowed entity size to be stored in the audit repo.
Diffs (updated)
-----
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java 958ecaf
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java 417092a
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 5099521
repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java cf76596
repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java 9f77bfe
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 9a9beff
Diff: https://reviews.apache.org/r/55592/diff/
Testing
-------
Tested using POSTMAN REST Client. Created large entities and made sure entities created fine.
Thanks,
Sarath Subramanian
Re: Review Request 55592: [ATLAS-1463] Mechanism to exclude specified
entity
attributes from getting stored in audit repository during entity
create/update/delete operations
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55592/#review161815
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java (line 299)
<https://reviews.apache.org/r/55592/#comment233120>
With addition of repositoryMaxSiz() in each repository implementation, this method may not be needed.
Also, consider moving getAuditExcludeAttributes() to specific repository implementation.
Please review.
- Madhan Neethiraj
On Jan. 16, 2017, 10:21 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55592/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2017, 10:21 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Suma Shivaprasad.
>
>
> Bugs: ATLAS-1463
> https://issues.apache.org/jira/browse/ATLAS-1463
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas uses HBase as its store for audit repository and when new entities are created the entity definition string along with all its attributes are stored in audit repository. When size of the entity is greater than "hbase.client.keyvalue.maxsize" property then entity creation fails with "KeyValue size too large" exception.
> A property needs to be set in atlas-application.properties to specify list of attributes to be excluded from getting stored in audit repo. Also check needs to be made to make sure entity size is not greater than keyvalue max size.
>
> This patch excludes specified attribute from getting stored in audit repository and also does a size check for max allowed entity size to be stored in the audit repo.
>
>
> Diffs
> -----
>
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java 958ecaf
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java 417092a
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 5099521
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java cf76596
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java 9f77bfe
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 9a9beff
>
> Diff: https://reviews.apache.org/r/55592/diff/
>
>
> Testing
> -------
>
> Tested using POSTMAN REST Client. Created large entities and made sure entities created fine.
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 55592: [ATLAS-1463] Mechanism to exclude specified
entity
attributes from getting stored in audit repository during entity
create/update/delete operations
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55592/#review161812
-----------------------------------------------------------
Fix it, then Ship it!
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java (line 115)
<https://reviews.apache.org/r/55592/#comment233118>
To keep the audit message same as earlier, pass "Deleted entity" as the the last argument; else createEvent() method will dump JSON string of the entity that was deleted.
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java (line 225)
<https://reviews.apache.org/r/55592/#comment233119>
Instead of returning a hardcoded value, read the valuue from configuration - "atlas.hbase.client.keyvalue.maxsize" (refer to EntityAuditListener.java).
- Madhan Neethiraj
On Jan. 16, 2017, 10:21 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55592/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2017, 10:21 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Suma Shivaprasad.
>
>
> Bugs: ATLAS-1463
> https://issues.apache.org/jira/browse/ATLAS-1463
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas uses HBase as its store for audit repository and when new entities are created the entity definition string along with all its attributes are stored in audit repository. When size of the entity is greater than "hbase.client.keyvalue.maxsize" property then entity creation fails with "KeyValue size too large" exception.
> A property needs to be set in atlas-application.properties to specify list of attributes to be excluded from getting stored in audit repo. Also check needs to be made to make sure entity size is not greater than keyvalue max size.
>
> This patch excludes specified attribute from getting stored in audit repository and also does a size check for max allowed entity size to be stored in the audit repo.
>
>
> Diffs
> -----
>
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java 958ecaf
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java 417092a
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 5099521
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java cf76596
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java 9f77bfe
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 9a9beff
>
> Diff: https://reviews.apache.org/r/55592/diff/
>
>
> Testing
> -------
>
> Tested using POSTMAN REST Client. Created large entities and made sure entities created fine.
>
>
> Thanks,
>
> Sarath Subramanian
>
>