You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2019/01/10 23:52:15 UTC
Review Request 69707: API: Audit Scan for Classification Updates
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/
-----------------------------------------------------------
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Bugs: ATLAS-3029
https://issues.apache.org/jira/browse/ATLAS-3029
Repository: atlas
Description
-------
**Approach**
- Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
- Package the output into _AtlasEntityHeader_.
- Provide _REST APIs_ to _EntityREST_.
**CURL**
_Fetch_
curl -X GET -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/audit/classifications?fromTimestamp=1546992000000&toTimestamp=1547157600000' \
_Udpate_
curl -X POST -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/audit/classifications?fromTimestamp=1546992000000&toTimestamp=1547157600000'
-d @fetched-classifications.json
**Not supported**
- Cassandra based Audits.
- For testing: In-memory audits.
Diffs
-----
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
intg/src/main/java/org/apache/atlas/model/audit/AuditActionEntityHeaderPair.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/audit/AuditActionScanResponse.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/audit/AuditRetrieveAndUpdate.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
repository/src/test/java/org/apache/atlas/repository/audit/AuditRetrieveAndUpdateTest.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
webapp/src/test/resources/json/audit-classifications.json PRE-CREATION
Diff: https://reviews.apache.org/r/69707/diff/1/
Testing
-------
**Functional tests**
Via CURL calls.
**Volume tests**
Loads of over 2000 entities.
File Attachments
----------------
fetched-classifications.json
https://reviews.apache.org/media/uploaded/files/2019/01/10/21bdf538-e1b5-4f1c-9130-ef5d0ee9a304__fetched-classifications.json
Thanks,
Ashutosh Mestry
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/#review211992
-----------------------------------------------------------
intg/src/main/java/org/apache/atlas/model/instance/ClassificationAssociateResponse.java
Lines 39 (patched)
<https://reviews.apache.org/r/69707/#comment297573>
ClassificationAssociateResponse ==> AtlasEntityHeaders
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java
Lines 132 (patched)
<https://reviews.apache.org/r/69707/#comment297590>
classifications that are already associated with the entity, but not in the incoming entity should be deleted as well i.e. not just when the incoming entity has empty classification list.
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Lines 95 (patched)
<https://reviews.apache.org/r/69707/#comment297592>
ClassificationAssociator is needed only in getEntityHeaders() method. Consider instantiating one in this method, instead of injecting here.
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Line 264 (original), 277 (patched)
<https://reviews.apache.org/r/69707/#comment297589>
perhaps this comment was updated during code-refactoring?
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Lines 759 (patched)
<https://reviews.apache.org/r/69707/#comment297591>
Consider adding AtlasEntityStore.setClassifications() method (similar to add/update/delete classification methods) and call it directly from here, instead of going through classificationAssociator.
- Madhan Neethiraj
On Jan. 15, 2019, 1:26 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69707/
> -----------------------------------------------------------
>
> (Updated Jan. 15, 2019, 1:26 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3029
> https://issues.apache.org/jira/browse/ATLAS-3029
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> - Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
> - Package the output into _AtlasEntityHeader_.
> - Provide _REST APIs_ to _EntityREST_.
>
> **CURL**
>
> _Fetch_
> curl -X GET -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/audit/classifications?timeRange=1546992000000:1547157600000' \
>
> _Udpate_
> curl -X POST -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/audit/classifications'
> -d @fetched-classifications.json
>
>
> **Not supported**
> - Cassandra based Audits.
> - For testing: In-memory audits.
>
>
> Diffs
> -----
>
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
> intg/src/main/java/org/apache/atlas/model/instance/ClassificationAssociateResponse.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java PRE-CREATION
> repository/src/test/resources/json/entity-headers.json PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
>
>
> Diff: https://reviews.apache.org/r/69707/diff/3/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Via CURL calls.
>
> **Volume tests**
> Loads of over 2000 entities.
>
>
> File Attachments
> ----------------
>
> entity-headers.json
> https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/#review212127
-----------------------------------------------------------
Fix it, then Ship it!
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java
Lines 60 (patched)
<https://reviews.apache.org/r/69707/#comment297750>
It might be simpler to add following method in EntityGraphRetriever directly, instead of a separate class ClassificationAssociator.
public class EntityGraphRetriever {
AtlasEntityHeaders getEntitiesWithTagChanges(fromTimestamp, toTimestamp) {
...
}
}
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java
Lines 80 (patched)
<https://reviews.apache.org/r/69707/#comment297746>
'Updater' class is no more used. If true, please remove.
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
Line 98 (original), 98 (patched)
<https://reviews.apache.org/r/69707/#comment297745>
Why remove 'final' here?
- Madhan Neethiraj
On Jan. 16, 2019, 5:40 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69707/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2019, 5:40 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3029
> https://issues.apache.org/jira/browse/ATLAS-3029
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> - Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
> - New: Package the output into _AtlasEntityHeaders_.
> - Provide _REST APIs_ to _EntityREST_.
> - _AtlasClientV2_ updated to support the new endpoints.
>
> **Rules for _setClassifications_**
> - If incoming _AtlasEntityHeader_:
> - Has no classifications: Delete entity's classification.
> - Has classifications & entity does not have the classification: Add to entity's classifications.
> - Has classifications & entity has the classification: Update entity's classification.
>
>
> **CURL**
>
> _Fetch_
> curl -X GET -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/bulk/headers?tagUpdateStartTime=1546992000000' \
>
> _Udpate_
> curl -X POST -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/bulk/setClassifications'
> -d @fetched-classifications.json
>
>
> **Not supported**
> - Cassandra based Audits.
> - For testing: In-memory audits.
>
>
> Diffs
> -----
>
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
> intg/src/main/java/org/apache/atlas/model/instance/AtlasEntityHeaders.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 750fa1775
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 35aa3afed
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc109
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-2-classification.json PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-2-classifications.json PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-2.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-1.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-2-classifications.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-2-no-classifications.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-2.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-3.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-empty.json PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
>
>
> Diff: https://reviews.apache.org/r/69707/diff/7/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Via CURL calls.
>
> **Volume tests**
> Loads of over 2000 entities.
>
>
> File Attachments
> ----------------
>
> entity-headers.json
> https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/#review212152
-----------------------------------------------------------
intg/src/main/java/org/apache/atlas/model/instance/AtlasEntityHeaders.java
Lines 40 (patched)
<https://reviews.apache.org/r/69707/#comment297803>
entityHeaderMap => guidHeaderMap
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java
Lines 111 (patched)
<https://reviews.apache.org/r/69707/#comment297802>
Instead of retrieving AtlasEntityWithExtInfo, consider the following:
AtlasEntityHeader exitingEntity = null;
AtlasVertex vertex = AtlasGraphUtilV2.findByUniqueAttributes(entityType, entityHeader.getAttributes());
if (vertex != null) {
existingEntity = entityRetriever.toAtlasEntityHeaderWithClassifications(vertex);
}
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java
Lines 127 (patched)
<https://reviews.apache.org/r/69707/#comment297807>
List<AtlasClassification> toAdd = new ArrayList<>();
List<AtlasClassification> toUpdate = new ArrayList<>();
List<AtlasClassification> toDelete = new ArrayList<>();
computeChanges(existingEntity.getClassifications(), incomingEntity.getClassifications(), toAdd, toUpdate, toDelete);
if (CollectionUtils.isNotEmpty(toAdd)) {
entityStore.addClassifications(existingEntity.getGuid(), toAdd);
}
if (CollectionUtils.isNotEmpty(toUpdate)) {
entityStore.updateClassifications(existingEntity.getGuid(), toUpdate);
}
if (CollectionUtils.isNotEmpty(toDelete)) {
for (AtlasClassification classification : toDelete) {
entityStore.deleteClassification(existingEntity.getGuid(), classification.getName());
}
}
- Madhan Neethiraj
On Jan. 18, 2019, 7:40 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69707/
> -----------------------------------------------------------
>
> (Updated Jan. 18, 2019, 7:40 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3029
> https://issues.apache.org/jira/browse/ATLAS-3029
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> - Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
> - New: Package the output into _AtlasEntityHeaders_.
> - Provide _REST APIs_ to _EntityREST_.
> - _AtlasClientV2_ updated to support the new endpoints.
>
> **Rules for _setClassifications_**
> - If incoming _AtlasEntityHeader_:
> - Has no classifications: Delete entity's classification.
> - Has classifications & entity does not have the classification: Add to entity's classifications.
> - Has classifications & entity has the classification: Update entity's classification.
>
>
> **CURL**
>
> _Fetch_
> curl -X GET -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/bulk/headers?tagUpdateStartTime=1546992000000' \
>
> _Udpate_
> curl -X POST -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/bulk/setClassifications'
> -d @fetched-classifications.json
>
>
> **Not supported**
> - Cassandra based Audits.
> - For testing: In-memory audits.
>
>
> Diffs
> -----
>
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
> intg/src/main/java/org/apache/atlas/model/instance/AtlasEntityHeaders.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 750fa1775
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 35aa3afed
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc109
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-2-classification.json PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-2-classifications.json PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-2.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-1.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-2-classifications.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-2-no-classifications.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-2.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-3.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-empty.json PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
>
>
> Diff: https://reviews.apache.org/r/69707/diff/7/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Via CURL calls.
>
> **Volume tests**
> Loads of over 2000 entities.
> Classification types: 50
> Entities: 8102
> File size: 3.3 MB.
> Duration: 11 mins
>
>
> File Attachments
> ----------------
>
> entity-headers.json
> https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/#review212172
-----------------------------------------------------------
Fix it, then Ship it!
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java
Lines 157 (patched)
<https://reviews.apache.org/r/69707/#comment297840>
If propagated classifications are present in 'incomingEntityHeader' and 'entityToBeUpdated', they should be ignored from add/update/delete operations. One option is for computeChanges() to ignore them.
- Madhan Neethiraj
On Jan. 20, 2019, 4:49 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69707/
> -----------------------------------------------------------
>
> (Updated Jan. 20, 2019, 4:49 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3029
> https://issues.apache.org/jira/browse/ATLAS-3029
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> - Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
> - New: Package the output into _AtlasEntityHeaders_.
> - Provide _REST APIs_ to _EntityREST_.
> - _AtlasClientV2_ updated to support the new endpoints.
>
> **Rules for _setClassifications_**
> - If incoming _AtlasEntityHeader_:
> - Has no classifications: Delete entity's classification.
> - Has classifications & entity does not have the classification: Add to entity's classifications.
> - Has classifications & entity has the classification: Update entity's classification.
>
>
> **CURL**
>
> _Fetch_
> curl -X GET -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/bulk/headers?tagUpdateStartTime=1546992000000' \
>
> _Udpate_
> curl -X POST -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/bulk/setClassifications'
> -d @fetched-classifications.json
>
>
> **Not supported**
> - Cassandra based Audits.
> - For testing: In-memory audits.
>
>
> Diffs
> -----
>
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
> intg/src/main/java/org/apache/atlas/model/instance/AtlasEntityHeaders.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 750fa1775
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 35aa3afed
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc109
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-2-classification.json PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-2-classifications.json PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-2.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-1.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-2-classifications.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-2-no-classifications.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-2.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-3.json PRE-CREATION
> repository/src/test/resources/json/classification-association/entity-header-empty.json PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
>
>
> Diff: https://reviews.apache.org/r/69707/diff/8/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Via CURL calls.
>
> **Volume tests**
> Loads of over 2000 entities.
> Classification types: 50
> Entities: 8102
> File size: 3.3 MB.
> Duration: 11 mins
>
>
> File Attachments
> ----------------
>
> entity-headers.json
> https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/#review212181
-----------------------------------------------------------
Fix it, then Ship it!
0001-ATLAS-3020-Audit-APIs-for-classification-updates.patch
Lines 1 (patched)
<https://reviews.apache.org/r/69707/#comment297846>
Patch file included by mistake? Please remove before committing the patch.
- Madhan Neethiraj
On Jan. 21, 2019, 7:59 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69707/
> -----------------------------------------------------------
>
> (Updated Jan. 21, 2019, 7:59 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3029
> https://issues.apache.org/jira/browse/ATLAS-3029
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> - Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
> - New: Package the output into _AtlasEntityHeaders_.
> - Provide _REST APIs_ to _EntityREST_.
> - _AtlasClientV2_ updated to support the new endpoints.
>
> **Rules for _setClassifications_**
> - If incoming _AtlasEntityHeader_:
> - Has no classifications: Delete entity's classification.
> - Has classifications & entity does not have the classification: Add to entity's classifications.
> - Has classifications & entity has the classification: Update entity's classification.
>
>
> **CURL**
>
> _Fetch_
> curl -X GET -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/bulk/headers?tagUpdateStartTime=1546992000000' \
>
> _Udpate_
> curl -X POST -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/bulk/setClassifications'
> -d @fetched-classifications.json
>
>
> **Not supported**
> - Cassandra based Audits.
> - For testing: In-memory audits.
>
>
> Diffs
> -----
>
> 0001-ATLAS-3020-Audit-APIs-for-classification-updates.patch PRE-CREATION
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
> intg/src/main/java/org/apache/atlas/model/instance/AtlasEntityHeaders.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 750fa1775
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 35aa3afed
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc109
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-None.json PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-PII-FIN_PII.json PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-PII.json PRE-CREATION
> repository/src/test/resources/json/classification-association/col-entity-T1-prop-Tn.json PRE-CREATION
> repository/src/test/resources/json/classification-association/header-FIN_PII.json PRE-CREATION
> repository/src/test/resources/json/classification-association/header-None.json PRE-CREATION
> repository/src/test/resources/json/classification-association/header-PII-VENDOR_PII.json PRE-CREATION
> repository/src/test/resources/json/classification-association/header-PII.json PRE-CREATION
> repository/src/test/resources/json/classification-association/header-Tx-prop-T1.json PRE-CREATION
> repository/src/test/resources/json/classification-association/header-Tx.json PRE-CREATION
> repository/src/test/resources/json/classification-association/header-empty.json PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
>
>
> Diff: https://reviews.apache.org/r/69707/diff/9/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Via CURL calls.
>
> **Volume tests**
> Loads of over 2000 entities.
> Classification types: 50
> Entities: 8102
> File size: 3.3 MB.
> Duration: 11 mins
>
>
> File Attachments
> ----------------
>
> entity-headers.json
> https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/
-----------------------------------------------------------
(Updated Jan. 21, 2019, 7:59 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include:
- Addressed review comments.
- Unit tests refactored for brevity.
Bugs: ATLAS-3029
https://issues.apache.org/jira/browse/ATLAS-3029
Repository: atlas
Description
-------
**Approach**
- Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
- New: Package the output into _AtlasEntityHeaders_.
- Provide _REST APIs_ to _EntityREST_.
- _AtlasClientV2_ updated to support the new endpoints.
**Rules for _setClassifications_**
- If incoming _AtlasEntityHeader_:
- Has no classifications: Delete entity's classification.
- Has classifications & entity does not have the classification: Add to entity's classifications.
- Has classifications & entity has the classification: Update entity's classification.
**CURL**
_Fetch_
curl -X GET -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/bulk/headers?tagUpdateStartTime=1546992000000' \
_Udpate_
curl -X POST -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/bulk/setClassifications'
-d @fetched-classifications.json
**Not supported**
- Cassandra based Audits.
- For testing: In-memory audits.
Diffs (updated)
-----
0001-ATLAS-3020-Audit-APIs-for-classification-updates.patch PRE-CREATION
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
intg/src/main/java/org/apache/atlas/model/instance/AtlasEntityHeaders.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 750fa1775
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 35aa3afed
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc109
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java PRE-CREATION
repository/src/test/resources/json/classification-association/col-entity-None.json PRE-CREATION
repository/src/test/resources/json/classification-association/col-entity-PII-FIN_PII.json PRE-CREATION
repository/src/test/resources/json/classification-association/col-entity-PII.json PRE-CREATION
repository/src/test/resources/json/classification-association/col-entity-T1-prop-Tn.json PRE-CREATION
repository/src/test/resources/json/classification-association/header-FIN_PII.json PRE-CREATION
repository/src/test/resources/json/classification-association/header-None.json PRE-CREATION
repository/src/test/resources/json/classification-association/header-PII-VENDOR_PII.json PRE-CREATION
repository/src/test/resources/json/classification-association/header-PII.json PRE-CREATION
repository/src/test/resources/json/classification-association/header-Tx-prop-T1.json PRE-CREATION
repository/src/test/resources/json/classification-association/header-Tx.json PRE-CREATION
repository/src/test/resources/json/classification-association/header-empty.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
Diff: https://reviews.apache.org/r/69707/diff/9/
Changes: https://reviews.apache.org/r/69707/diff/8-9/
Testing
-------
**Unit tests**
New tests added.
**Functional tests**
Via CURL calls.
**Volume tests**
Loads of over 2000 entities.
Classification types: 50
Entities: 8102
File size: 3.3 MB.
Duration: 11 mins
File Attachments
----------------
entity-headers.json
https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
Thanks,
Ashutosh Mestry
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/
-----------------------------------------------------------
(Updated Jan. 20, 2019, 4:49 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include:
- Addressed review comments.
- Additional refactoring.
- Additional unit tests.
Bugs: ATLAS-3029
https://issues.apache.org/jira/browse/ATLAS-3029
Repository: atlas
Description
-------
**Approach**
- Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
- New: Package the output into _AtlasEntityHeaders_.
- Provide _REST APIs_ to _EntityREST_.
- _AtlasClientV2_ updated to support the new endpoints.
**Rules for _setClassifications_**
- If incoming _AtlasEntityHeader_:
- Has no classifications: Delete entity's classification.
- Has classifications & entity does not have the classification: Add to entity's classifications.
- Has classifications & entity has the classification: Update entity's classification.
**CURL**
_Fetch_
curl -X GET -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/bulk/headers?tagUpdateStartTime=1546992000000' \
_Udpate_
curl -X POST -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/bulk/setClassifications'
-d @fetched-classifications.json
**Not supported**
- Cassandra based Audits.
- For testing: In-memory audits.
Diffs (updated)
-----
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
intg/src/main/java/org/apache/atlas/model/instance/AtlasEntityHeaders.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 750fa1775
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 35aa3afed
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc109
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java PRE-CREATION
repository/src/test/resources/json/classification-association/col-entity-2-classification.json PRE-CREATION
repository/src/test/resources/json/classification-association/col-entity-2-classifications.json PRE-CREATION
repository/src/test/resources/json/classification-association/col-entity-2.json PRE-CREATION
repository/src/test/resources/json/classification-association/entity-header-1.json PRE-CREATION
repository/src/test/resources/json/classification-association/entity-header-2-classifications.json PRE-CREATION
repository/src/test/resources/json/classification-association/entity-header-2-no-classifications.json PRE-CREATION
repository/src/test/resources/json/classification-association/entity-header-2.json PRE-CREATION
repository/src/test/resources/json/classification-association/entity-header-3.json PRE-CREATION
repository/src/test/resources/json/classification-association/entity-header-empty.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
Diff: https://reviews.apache.org/r/69707/diff/8/
Changes: https://reviews.apache.org/r/69707/diff/7-8/
Testing
-------
**Unit tests**
New tests added.
**Functional tests**
Via CURL calls.
**Volume tests**
Loads of over 2000 entities.
Classification types: 50
Entities: 8102
File size: 3.3 MB.
Duration: 11 mins
File Attachments
----------------
entity-headers.json
https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
Thanks,
Ashutosh Mestry
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/
-----------------------------------------------------------
(Updated Jan. 18, 2019, 7:40 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include: Added volume testing results.
Bugs: ATLAS-3029
https://issues.apache.org/jira/browse/ATLAS-3029
Repository: atlas
Description
-------
**Approach**
- Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
- New: Package the output into _AtlasEntityHeaders_.
- Provide _REST APIs_ to _EntityREST_.
- _AtlasClientV2_ updated to support the new endpoints.
**Rules for _setClassifications_**
- If incoming _AtlasEntityHeader_:
- Has no classifications: Delete entity's classification.
- Has classifications & entity does not have the classification: Add to entity's classifications.
- Has classifications & entity has the classification: Update entity's classification.
**CURL**
_Fetch_
curl -X GET -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/bulk/headers?tagUpdateStartTime=1546992000000' \
_Udpate_
curl -X POST -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/bulk/setClassifications'
-d @fetched-classifications.json
**Not supported**
- Cassandra based Audits.
- For testing: In-memory audits.
Diffs
-----
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
intg/src/main/java/org/apache/atlas/model/instance/AtlasEntityHeaders.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 750fa1775
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 35aa3afed
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc109
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java PRE-CREATION
repository/src/test/resources/json/classification-association/col-entity-2-classification.json PRE-CREATION
repository/src/test/resources/json/classification-association/col-entity-2-classifications.json PRE-CREATION
repository/src/test/resources/json/classification-association/col-entity-2.json PRE-CREATION
repository/src/test/resources/json/classification-association/entity-header-1.json PRE-CREATION
repository/src/test/resources/json/classification-association/entity-header-2-classifications.json PRE-CREATION
repository/src/test/resources/json/classification-association/entity-header-2-no-classifications.json PRE-CREATION
repository/src/test/resources/json/classification-association/entity-header-2.json PRE-CREATION
repository/src/test/resources/json/classification-association/entity-header-3.json PRE-CREATION
repository/src/test/resources/json/classification-association/entity-header-empty.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
Diff: https://reviews.apache.org/r/69707/diff/7/
Testing (updated)
-------
**Unit tests**
New tests added.
**Functional tests**
Via CURL calls.
**Volume tests**
Loads of over 2000 entities.
Classification types: 50
Entities: 8102
File size: 3.3 MB.
Duration: 11 mins
File Attachments
----------------
entity-headers.json
https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
Thanks,
Ashutosh Mestry
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/
-----------------------------------------------------------
(Updated Jan. 16, 2019, 5:40 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include: Addressed review comments.
Bugs: ATLAS-3029
https://issues.apache.org/jira/browse/ATLAS-3029
Repository: atlas
Description
-------
**Approach**
- Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
- Package the output into _AtlasEntityHeader_.
- Provide _REST APIs_ to _EntityREST_.
**CURL**
_Fetch_
curl -X GET -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/bulk/headers?timeRange=1546992000000:1547157600000' \
_Udpate_
curl -X POST -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/bulk/setClassifications'
-d @fetched-classifications.json
**Not supported**
- Cassandra based Audits.
- For testing: In-memory audits.
Diffs (updated)
-----
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
intg/src/main/java/org/apache/atlas/model/instance/AtlasEntityHeaders.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 750fa1775
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 35aa3afed
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java PRE-CREATION
repository/src/test/resources/json/entity-headers.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
Diff: https://reviews.apache.org/r/69707/diff/5/
Changes: https://reviews.apache.org/r/69707/diff/4-5/
Testing
-------
**Unit tests**
New tests added.
**Functional tests**
Via CURL calls.
**Volume tests**
Loads of over 2000 entities.
File Attachments
----------------
entity-headers.json
https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
Thanks,
Ashutosh Mestry
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/
-----------------------------------------------------------
(Updated Jan. 16, 2019, 6:25 a.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include: Addressed review comments.
Bugs: ATLAS-3029
https://issues.apache.org/jira/browse/ATLAS-3029
Repository: atlas
Description
-------
**Approach**
- Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
- Package the output into _AtlasEntityHeader_.
- Provide _REST APIs_ to _EntityREST_.
**CURL**
_Fetch_
curl -X GET -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/bulk/headers?timeRange=1546992000000:1547157600000' \
_Udpate_
curl -X POST -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/bulk/setClassifications'
-d @fetched-classifications.json
**Not supported**
- Cassandra based Audits.
- For testing: In-memory audits.
Diffs (updated)
-----
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
intg/src/main/java/org/apache/atlas/model/instance/AtlasEntityHeaders.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 750fa1775
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 35aa3afed
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java PRE-CREATION
repository/src/test/resources/json/entity-headers.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
Diff: https://reviews.apache.org/r/69707/diff/4/
Changes: https://reviews.apache.org/r/69707/diff/3-4/
Testing
-------
**Unit tests**
New tests added.
**Functional tests**
Via CURL calls.
**Volume tests**
Loads of over 2000 entities.
File Attachments
----------------
entity-headers.json
https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
Thanks,
Ashutosh Mestry
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/
-----------------------------------------------------------
(Updated Jan. 15, 2019, 1:26 a.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include: Addressed review comments.
Bugs: ATLAS-3029
https://issues.apache.org/jira/browse/ATLAS-3029
Repository: atlas
Description
-------
**Approach**
- Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
- Package the output into _AtlasEntityHeader_.
- Provide _REST APIs_ to _EntityREST_.
**CURL**
_Fetch_
curl -X GET -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/audit/classifications?timeRange=1546992000000:1547157600000' \
_Udpate_
curl -X POST -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/audit/classifications'
-d @fetched-classifications.json
**Not supported**
- Cassandra based Audits.
- For testing: In-memory audits.
Diffs (updated)
-----
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
intg/src/main/java/org/apache/atlas/model/instance/ClassificationAssociateResponse.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java PRE-CREATION
repository/src/test/resources/json/entity-headers.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
Diff: https://reviews.apache.org/r/69707/diff/3/
Changes: https://reviews.apache.org/r/69707/diff/2-3/
Testing
-------
**Unit tests**
New tests added.
**Functional tests**
Via CURL calls.
**Volume tests**
Loads of over 2000 entities.
File Attachments
----------------
entity-headers.json
https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
Thanks,
Ashutosh Mestry
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On Jan. 14, 2019, 6:17 a.m., Madhan Neethiraj wrote:
> > intg/src/main/java/org/apache/atlas/model/instance/BulkHeadersResponse.java
> > Lines 40 (patched)
> > <https://reviews.apache.org/r/69707/diff/2/?file=2119437#file2119437line40>
> >
> > Please review if BulkHeadersResponse class is necessary; perhaps the return value can simply be Map<String, AtlasEntityHeader>?
> >
> > The interpretation of 'fromTimestamp' and 'toTimestamp' attributes here is not clear; also, more criterion can be added in the REST API, and we can avoid having to update BulkHeadersResponse to be in sync.
The Map<String, AtlasEntityHeader> ends with the serialization problem. We will have to use _TypeRefernce_ within _AtlasClient_ methods. Hence I choose this.
> On Jan. 14, 2019, 6:17 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
> > Lines 87 (patched)
> > <https://reviews.apache.org/r/69707/diff/2/?file=2119440#file2119440line87>
> >
> > Instead of returning List<EntityAuditEventV2>, returning Set<String> i.e. guids might be more efficient; contents of EntityAuditEventV2 is not useful for the current use of this method. Please reveiew.
This is the change in approach I mentioned earlier. Just returnining guids will not make us get the state during that time range. We will end up with the latest state. Hence the events are returned.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/#review211944
-----------------------------------------------------------
On Jan. 14, 2019, 5:23 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69707/
> -----------------------------------------------------------
>
> (Updated Jan. 14, 2019, 5:23 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3029
> https://issues.apache.org/jira/browse/ATLAS-3029
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> - Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
> - Package the output into _AtlasEntityHeader_.
> - Provide _REST APIs_ to _EntityREST_.
>
> **CURL**
>
> _Fetch_
> curl -X GET -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/audit/classifications?timeRange=1546992000000:1547157600000' \
>
> _Udpate_
> curl -X POST -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/audit/classifications'
> -d @fetched-classifications.json
>
>
> **Not supported**
> - Cassandra based Audits.
> - For testing: In-memory audits.
>
>
> Diffs
> -----
>
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
> intg/src/main/java/org/apache/atlas/model/instance/BulkHeadersResponse.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/audit/BulkEntityRetrieveAndUpdate.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
> repository/src/test/java/org/apache/atlas/repository/audit/BulkEntityRetrieveAndUpdateTest.java PRE-CREATION
> repository/src/test/resources/json/audit-event-1.json PRE-CREATION
> repository/src/test/resources/json/audit-event-2.json PRE-CREATION
> repository/src/test/resources/json/entity-headers.json PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
>
>
> Diff: https://reviews.apache.org/r/69707/diff/2/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Via CURL calls.
>
> **Volume tests**
> Loads of over 2000 entities.
>
>
> File Attachments
> ----------------
>
> entity-headers.json
> https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/#review211944
-----------------------------------------------------------
intg/src/main/java/org/apache/atlas/model/instance/BulkHeadersResponse.java
Lines 40 (patched)
<https://reviews.apache.org/r/69707/#comment297512>
Please review if BulkHeadersResponse class is necessary; perhaps the return value can simply be Map<String, AtlasEntityHeader>?
The interpretation of 'fromTimestamp' and 'toTimestamp' attributes here is not clear; also, more criterion can be added in the REST API, and we can avoid having to update BulkHeadersResponse to be in sync.
repository/src/main/java/org/apache/atlas/repository/audit/BulkEntityRetrieveAndUpdate.java
Lines 82 (patched)
<https://reviews.apache.org/r/69707/#comment297513>
Consider replacing this method with the following:
Map<String, AtlasEntityHeader> ret = new HashMap();
Set<String> guids = auditRepository.getEntitiesWithTagChanges(fromTimestamp, toTimestamp);
for (String guid : guids) {
AtlasEntityHeader header = entityRetriever.toAtlasEntityHeaderWithClassifications(entityId);
ret.put(guid, header);
}
return ret;
repository/src/main/java/org/apache/atlas/repository/audit/BulkEntityRetrieveAndUpdate.java
Lines 142 (patched)
<https://reviews.apache.org/r/69707/#comment297506>
'Updater' class is not related to 'audit' module; hence consider moving this to EntityStore, with the following signature:
- setClassifications(Map<String, AtlasEntityHeader> entities)
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
Lines 87 (patched)
<https://reviews.apache.org/r/69707/#comment297511>
Instead of returning List<EntityAuditEventV2>, returning Set<String> i.e. guids might be more efficient; contents of EntityAuditEventV2 is not useful for the current use of this method. Please reveiew.
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Lines 729 (patched)
<https://reviews.apache.org/r/69707/#comment297507>
timeRange => tagUpdateTimeRange
getEntitiesWithModifiedClassifications() ==> getEntityHeaders()
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Lines 751 (patched)
<https://reviews.apache.org/r/69707/#comment297509>
Following private methods don't seem to be used; please review and remove:
- getClassificationName() - #751
- incrementToTimestamp() - #755
- getEntityById() - #777
- getClassification() - #786
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Lines 760 (patched)
<https://reviews.apache.org/r/69707/#comment297508>
"bulk/headers" ==> "bulk/setClassifications"
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Lines 763 (patched)
<https://reviews.apache.org/r/69707/#comment297510>
updateEntitiesWithModifiedClassifications() ==> setClassifications()
- Madhan Neethiraj
On Jan. 14, 2019, 5:23 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69707/
> -----------------------------------------------------------
>
> (Updated Jan. 14, 2019, 5:23 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3029
> https://issues.apache.org/jira/browse/ATLAS-3029
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> - Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
> - Package the output into _AtlasEntityHeader_.
> - Provide _REST APIs_ to _EntityREST_.
>
> **CURL**
>
> _Fetch_
> curl -X GET -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/audit/classifications?timeRange=1546992000000:1547157600000' \
>
> _Udpate_
> curl -X POST -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/audit/classifications'
> -d @fetched-classifications.json
>
>
> **Not supported**
> - Cassandra based Audits.
> - For testing: In-memory audits.
>
>
> Diffs
> -----
>
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
> intg/src/main/java/org/apache/atlas/model/instance/BulkHeadersResponse.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/audit/BulkEntityRetrieveAndUpdate.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
> repository/src/test/java/org/apache/atlas/repository/audit/BulkEntityRetrieveAndUpdateTest.java PRE-CREATION
> repository/src/test/resources/json/audit-event-1.json PRE-CREATION
> repository/src/test/resources/json/audit-event-2.json PRE-CREATION
> repository/src/test/resources/json/entity-headers.json PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
>
>
> Diff: https://reviews.apache.org/r/69707/diff/2/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Via CURL calls.
>
> **Volume tests**
> Loads of over 2000 entities.
>
>
> File Attachments
> ----------------
>
> entity-headers.json
> https://reviews.apache.org/media/uploaded/files/2019/01/14/a23bead7-6990-4d5b-b3b8-f46f990f5d99__entity-headers.json
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/
-----------------------------------------------------------
(Updated Jan. 14, 2019, 5:23 a.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include:
- Modified approach for retrieval.
- Replaced data structures with simpler version.
- Updated REST APIs.
Bugs: ATLAS-3029
https://issues.apache.org/jira/browse/ATLAS-3029
Repository: atlas
Description (updated)
-------
**Approach**
- Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
- Package the output into _AtlasEntityHeader_.
- Provide _REST APIs_ to _EntityREST_.
**CURL**
_Fetch_
curl -X GET -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/audit/classifications?timeRange=1546992000000:1547157600000' \
_Udpate_
curl -X POST -u admin:passWord -H 'Content-Type: application/json'
'http://localhost:21000/api/atlas/v2/entity/audit/classifications'
-d @fetched-classifications.json
**Not supported**
- Cassandra based Audits.
- For testing: In-memory audits.
Diffs (updated)
-----
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
intg/src/main/java/org/apache/atlas/model/instance/BulkHeadersResponse.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/audit/BulkEntityRetrieveAndUpdate.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
repository/src/test/java/org/apache/atlas/repository/audit/BulkEntityRetrieveAndUpdateTest.java PRE-CREATION
repository/src/test/resources/json/audit-event-1.json PRE-CREATION
repository/src/test/resources/json/audit-event-2.json PRE-CREATION
repository/src/test/resources/json/entity-headers.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
Diff: https://reviews.apache.org/r/69707/diff/2/
Changes: https://reviews.apache.org/r/69707/diff/1-2/
Testing (updated)
-------
**Unit tests**
New tests added.
**Functional tests**
Via CURL calls.
**Volume tests**
Loads of over 2000 entities.
Thanks,
Ashutosh Mestry
Re: Review Request 69707: API: Audit Scan for Classification Updates
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69707/#review211906
-----------------------------------------------------------
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Lines 730 (patched)
<https://reviews.apache.org/r/69707/#comment297477>
Consider the following signature for this method:
@GET
@Path("/bulk/headers")
Map<String, AtlasEntityHeader> getEntities(@QueryParam("tagUpdateTime") String tagUpdateTime) {
int sepPos = tagUpdateTime.indexOf(":");
String tagUpdateStartTime = tagUpdateTime.substring(0, sepPos);
String tagUpdateEndTime = tagUpdateTime.substring(sepPos + 1);
Map<String, AtlasEntityHeader> ret = new HashMap<>();
// 1. scan audit table to retrieve guids of entities having classification changes in the given duration
// 2. for each guid retrieve, populate 'ret' with current state of the entity - along with classification details
return ret;
}
This method can further be enhanced to include more criterion - like 'createTime', 'modifiedTime', 'entityType'
- Madhan Neethiraj
On Jan. 10, 2019, 11:52 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69707/
> -----------------------------------------------------------
>
> (Updated Jan. 10, 2019, 11:52 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3029
> https://issues.apache.org/jira/browse/ATLAS-3029
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> - Scan HBase 'ATLAS_ENTITY_AUDIT_EVENTS' table with approprite paramters.
> - Package the output into _AtlasEntityHeader_.
> - Provide _REST APIs_ to _EntityREST_.
>
> **CURL**
>
> _Fetch_
> curl -X GET -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/audit/classifications?fromTimestamp=1546992000000&toTimestamp=1547157600000' \
>
> _Udpate_
> curl -X POST -u admin:passWord -H 'Content-Type: application/json'
> 'http://localhost:21000/api/atlas/v2/entity/audit/classifications?fromTimestamp=1546992000000&toTimestamp=1547157600000'
> -d @fetched-classifications.json
>
>
> **Not supported**
> - Cassandra based Audits.
> - For testing: In-memory audits.
>
>
> Diffs
> -----
>
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7c8caee89
> intg/src/main/java/org/apache/atlas/model/audit/AuditActionEntityHeaderPair.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/audit/AuditActionScanResponse.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/audit/AuditRetrieveAndUpdate.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java eb78f8f70
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java aab2d5b6c
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java 6f4415fa8
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java dca3b853e
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java e3a607809
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRetrieveAndUpdateTest.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 68c132c37
> webapp/src/test/resources/json/audit-classifications.json PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/69707/diff/1/
>
>
> Testing
> -------
>
> **Functional tests**
> Via CURL calls.
>
> **Volume tests**
> Loads of over 2000 entities.
>
>
> File Attachments
> ----------------
>
> fetched-classifications.json
> https://reviews.apache.org/media/uploaded/files/2019/01/10/21bdf538-e1b5-4f1c-9130-ef5d0ee9a304__fetched-classifications.json
>
>
> Thanks,
>
> Ashutosh Mestry
>
>