You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Sidharth Mishra <si...@gmail.com> on 2020/02/25 22:14:49 UTC
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for
Atlas Purge Entity
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/
-----------------------------------------------------------
(Updated Feb. 25, 2020, 10:14 p.m.)
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
Bugs: ATLAS-3518
https://issues.apache.org/jira/browse/ATLAS-3518
Repository: atlas
Description
-------
Right now for purge, delete entity, etc. at Atlas we create audit entry at HBase. User can go to entity and click on audit to see the audit information for that particular entity. But if user purge one entity there will not be any vertex at Janus graph and user will not able to get the purge audit which is important for governance purpose.
As part of fix, implemented Atlas Audit Framework which is being used for Purge operation and can be extended for other operations. In Future once we change the AtlasAudit type as super type for Import export as well we can use this framework for all Atlas operations.
admin/audit rest has been added which will return all the Atlas audit operations including import export at Atlas.
Diffs (updated)
-----
addons/models/0000-Area0/0010-base_model.json 001bb6cfa
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c
dashboardv2/public/js/utils/Enums.js 82e11ca6c
dashboardv3/public/js/utils/Enums.js 82e11ca6c
intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java b448d510b
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java a87e2b57f
repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187
repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java be1698fb4
repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369
repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java 78865c569
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java ba7a8a04b
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 925b2a007
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 4d438525a
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java db8c6c83c
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 25e0a5362
repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java 1863b8d74
repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java d0c06a107
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 759be64bb
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 78fdaca8d
repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java 42b63535b
repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java a1b512fb9
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 7a1ed18cb
repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java 48b958271
repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java c9bb11c56
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 27a6668a5
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 18e950e8c
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java 050bda3ae
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java cbac18c8c
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java 6269d3f74
repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java e56e48308
repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java baeafd4b7
repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java PRE-CREATION
repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-purge.json PRE-CREATION
repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION
server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc
webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328
webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64
webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION
Diff: https://reviews.apache.org/r/71791/diff/15/
Changes: https://reviews.apache.org/r/71791/diff/14-15/
Testing
-------
Two UT and one IT has been added to test functionality like:
1. Atlas Audit entry save and retrive
2. Atlas Purge Audit store, retrive and search (UT and IT)
Thanks,
Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for
Atlas Purge Entity
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review219723
-----------------------------------------------------------
repository/src/test/java/org/apache/atlas/TestModules.java
Line 44 (original), 44 (patched)
<https://reviews.apache.org/r/71791/#comment307926>
Avoid * imports. You can change IntelliJ setting to remove this refactoring.
repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java
Line 96 (original), 96 (patched)
<https://reviews.apache.org/r/71791/#comment307927>
Avoid this.
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
Line 38 (original), 38 (patched)
<https://reviews.apache.org/r/71791/#comment307928>
Remove auto expand.
- Ashutosh Mestry
On Feb. 26, 2020, 9:21 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2020, 9:21 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Right now for purge, delete entity, etc. at Atlas we create audit entry at HBase. User can go to entity and click on audit to see the audit information for that particular entity. But if user purge one entity there will not be any vertex at Janus graph and user will not able to get the purge audit which is important for governance purpose.
>
> As part of fix, implemented Atlas Audit Framework which is being used for Purge operation and can be extended for other operations. In Future once we change the AtlasAudit type as super type for Import export as well we can use this framework for all Atlas operations.
>
> admin/audit rest has been added which will return all the Atlas audit operations including import export at Atlas.
>
>
> Diffs
> -----
>
> addons/models/0000-Area0/0010-base_model.json 6bdd2f7ba
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c
> dashboardv2/public/js/utils/Enums.js 82e11ca6c
> dashboardv3/public/js/utils/Enums.js 82e11ca6c
> intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java b448d510b
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java a87e2b57f
> repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187
> repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java be1698fb4
> repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java 78865c569
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java ba7a8a04b
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 925b2a007
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 4d438525a
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java db8c6c83c
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 25e0a5362
> repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java 1863b8d74
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java d0c06a107
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 78fdaca8d
> repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java 42b63535b
> repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java a1b512fb9
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 7a1ed18cb
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java 48b958271
> repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java c9bb11c56
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 0ffc3d595
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 18e950e8c
> repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java 050bda3ae
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java cbac18c8c
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java 6269d3f74
> repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java e56e48308
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java baeafd4b7
> repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java PRE-CREATION
> repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-purge.json PRE-CREATION
> repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION
> server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc
> webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49
> webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64
> webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/71791/diff/16/
>
>
> Testing
> -------
>
> Two UT and one IT has been added to test functionality like:
> 1. Atlas Audit entry save and retrive
> 2. Atlas Purge Audit store, retrive and search (UT and IT)
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for
Atlas Purge Entity
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review219727
-----------------------------------------------------------
Ship it!
Ship It!
- Sarath Subramanian
On March 2, 2020, 1:47 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> -----------------------------------------------------------
>
> (Updated March 2, 2020, 1:47 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Right now for purge, delete entity, etc. at Atlas we create audit entry at HBase. User can go to entity and click on audit to see the audit information for that particular entity. But if user purge one entity there will not be any vertex at Janus graph and user will not able to get the purge audit which is important for governance purpose.
>
> As part of fix, implemented Atlas Audit Framework which is being used for Purge operation and can be extended for other operations. In Future once we change the AtlasAudit type as super type for Import export as well we can use this framework for all Atlas operations.
>
> admin/audit rest has been added which will return all the Atlas audit operations including import export at Atlas.
>
>
> Diffs
> -----
>
> addons/models/0000-Area0/0010-base_model.json 6bdd2f7ba
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c
> dashboardv2/public/js/utils/Enums.js 82e11ca6c
> dashboardv3/public/js/utils/Enums.js 82e11ca6c
> intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java b448d510b
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java a87e2b57f
> repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187
> repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java be1698fb4
> repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java 78865c569
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java ba7a8a04b
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 925b2a007
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 4d438525a
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java db8c6c83c
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 25e0a5362
> repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java 1863b8d74
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java d0c06a107
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 116ffa747
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 78fdaca8d
> repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java 42b63535b
> repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java a1b512fb9
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 7a1ed18cb
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java 48b958271
> repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java c9bb11c56
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 0ffc3d595
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 18e950e8c
> repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java 050bda3ae
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java cbac18c8c
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java 6269d3f74
> repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java e56e48308
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java baeafd4b7
> repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java PRE-CREATION
> repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-purge.json PRE-CREATION
> repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION
> server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc
> webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49
> webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64
> webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/71791/diff/18/
>
>
> Testing
> -------
>
> Two UT and one IT has been added to test functionality like:
> 1. Atlas Audit entry save and retrive
> 2. Atlas Purge Audit store, retrive and search (UT and IT)
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for
Atlas Purge Entity
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/
-----------------------------------------------------------
(Updated March 2, 2020, 9:47 p.m.)
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
Bugs: ATLAS-3518
https://issues.apache.org/jira/browse/ATLAS-3518
Repository: atlas
Description
-------
Right now for purge, delete entity, etc. at Atlas we create audit entry at HBase. User can go to entity and click on audit to see the audit information for that particular entity. But if user purge one entity there will not be any vertex at Janus graph and user will not able to get the purge audit which is important for governance purpose.
As part of fix, implemented Atlas Audit Framework which is being used for Purge operation and can be extended for other operations. In Future once we change the AtlasAudit type as super type for Import export as well we can use this framework for all Atlas operations.
admin/audit rest has been added which will return all the Atlas audit operations including import export at Atlas.
Diffs (updated)
-----
addons/models/0000-Area0/0010-base_model.json 6bdd2f7ba
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c
dashboardv2/public/js/utils/Enums.js 82e11ca6c
dashboardv3/public/js/utils/Enums.js 82e11ca6c
intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java b448d510b
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java a87e2b57f
repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187
repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java be1698fb4
repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369
repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java 78865c569
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java ba7a8a04b
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 925b2a007
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 4d438525a
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java db8c6c83c
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 25e0a5362
repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java 1863b8d74
repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java d0c06a107
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 116ffa747
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 78fdaca8d
repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java 42b63535b
repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java a1b512fb9
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 7a1ed18cb
repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java 48b958271
repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java c9bb11c56
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 0ffc3d595
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 18e950e8c
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java 050bda3ae
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java cbac18c8c
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java 6269d3f74
repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java e56e48308
repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java baeafd4b7
repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java PRE-CREATION
repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-purge.json PRE-CREATION
repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION
server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc
webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328
webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64
webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION
Diff: https://reviews.apache.org/r/71791/diff/18/
Changes: https://reviews.apache.org/r/71791/diff/17-18/
Testing
-------
Two UT and one IT has been added to test functionality like:
1. Atlas Audit entry save and retrive
2. Atlas Purge Audit store, retrive and search (UT and IT)
Thanks,
Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for
Atlas Purge Entity
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/
-----------------------------------------------------------
(Updated March 2, 2020, 9:42 p.m.)
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
Bugs: ATLAS-3518
https://issues.apache.org/jira/browse/ATLAS-3518
Repository: atlas
Description
-------
Right now for purge, delete entity, etc. at Atlas we create audit entry at HBase. User can go to entity and click on audit to see the audit information for that particular entity. But if user purge one entity there will not be any vertex at Janus graph and user will not able to get the purge audit which is important for governance purpose.
As part of fix, implemented Atlas Audit Framework which is being used for Purge operation and can be extended for other operations. In Future once we change the AtlasAudit type as super type for Import export as well we can use this framework for all Atlas operations.
admin/audit rest has been added which will return all the Atlas audit operations including import export at Atlas.
Diffs (updated)
-----
addons/models/0000-Area0/0010-base_model.json 6bdd2f7ba
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c
dashboardv2/public/js/utils/Enums.js 82e11ca6c
dashboardv3/public/js/utils/Enums.js 82e11ca6c
intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java b448d510b
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java a87e2b57f
repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187
repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java be1698fb4
repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369
repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java 78865c569
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java ba7a8a04b
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 925b2a007
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 4d438525a
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java db8c6c83c
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 25e0a5362
repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java 1863b8d74
repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java d0c06a107
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 116ffa747
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 78fdaca8d
repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java 42b63535b
repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java a1b512fb9
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 7a1ed18cb
repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java 48b958271
repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java c9bb11c56
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 0ffc3d595
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 18e950e8c
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java 050bda3ae
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java cbac18c8c
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java 6269d3f74
repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java e56e48308
repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java baeafd4b7
repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java PRE-CREATION
repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-purge.json PRE-CREATION
repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION
server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc
webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328
webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64
webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION
Diff: https://reviews.apache.org/r/71791/diff/17/
Changes: https://reviews.apache.org/r/71791/diff/16-17/
Testing
-------
Two UT and one IT has been added to test functionality like:
1. Atlas Audit entry save and retrive
2. Atlas Purge Audit store, retrive and search (UT and IT)
Thanks,
Sidharth Mishra
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for
Atlas Purge Entity
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review219725
-----------------------------------------------------------
Ship it!
Fix it.
- Ashutosh Mestry
On Feb. 26, 2020, 9:21 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2020, 9:21 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Right now for purge, delete entity, etc. at Atlas we create audit entry at HBase. User can go to entity and click on audit to see the audit information for that particular entity. But if user purge one entity there will not be any vertex at Janus graph and user will not able to get the purge audit which is important for governance purpose.
>
> As part of fix, implemented Atlas Audit Framework which is being used for Purge operation and can be extended for other operations. In Future once we change the AtlasAudit type as super type for Import export as well we can use this framework for all Atlas operations.
>
> admin/audit rest has been added which will return all the Atlas audit operations including import export at Atlas.
>
>
> Diffs
> -----
>
> addons/models/0000-Area0/0010-base_model.json 6bdd2f7ba
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c
> dashboardv2/public/js/utils/Enums.js 82e11ca6c
> dashboardv3/public/js/utils/Enums.js 82e11ca6c
> intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java b448d510b
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java a87e2b57f
> repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187
> repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java be1698fb4
> repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java 78865c569
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java ba7a8a04b
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 925b2a007
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 4d438525a
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java db8c6c83c
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 25e0a5362
> repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java 1863b8d74
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java d0c06a107
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 78fdaca8d
> repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java 42b63535b
> repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java a1b512fb9
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 7a1ed18cb
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java 48b958271
> repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java c9bb11c56
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 0ffc3d595
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 18e950e8c
> repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java 050bda3ae
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java cbac18c8c
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java 6269d3f74
> repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java e56e48308
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java baeafd4b7
> repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java PRE-CREATION
> repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-purge.json PRE-CREATION
> repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION
> server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc
> webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49
> webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64
> webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/71791/diff/16/
>
>
> Testing
> -------
>
> Two UT and one IT has been added to test functionality like:
> 1. Atlas Audit entry save and retrive
> 2. Atlas Purge Audit store, retrive and search (UT and IT)
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for
Atlas Purge Entity
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/
-----------------------------------------------------------
(Updated Feb. 26, 2020, 9:21 p.m.)
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
Bugs: ATLAS-3518
https://issues.apache.org/jira/browse/ATLAS-3518
Repository: atlas
Description
-------
Right now for purge, delete entity, etc. at Atlas we create audit entry at HBase. User can go to entity and click on audit to see the audit information for that particular entity. But if user purge one entity there will not be any vertex at Janus graph and user will not able to get the purge audit which is important for governance purpose.
As part of fix, implemented Atlas Audit Framework which is being used for Purge operation and can be extended for other operations. In Future once we change the AtlasAudit type as super type for Import export as well we can use this framework for all Atlas operations.
admin/audit rest has been added which will return all the Atlas audit operations including import export at Atlas.
Diffs (updated)
-----
addons/models/0000-Area0/0010-base_model.json 6bdd2f7ba
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c
dashboardv2/public/js/utils/Enums.js 82e11ca6c
dashboardv3/public/js/utils/Enums.js 82e11ca6c
intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java b448d510b
repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java a87e2b57f
repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187
repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java be1698fb4
repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369
repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java 78865c569
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java ba7a8a04b
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 925b2a007
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 4d438525a
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java db8c6c83c
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 25e0a5362
repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java 1863b8d74
repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java d0c06a107
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 78fdaca8d
repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java 42b63535b
repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java a1b512fb9
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 7a1ed18cb
repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java 48b958271
repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java c9bb11c56
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 0ffc3d595
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 18e950e8c
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java 050bda3ae
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java cbac18c8c
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java 6269d3f74
repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java e56e48308
repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java baeafd4b7
repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java PRE-CREATION
repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-purge.json PRE-CREATION
repository/src/test/resources/json/auditSearchParameters/audit-search-parameter-without-filter.json PRE-CREATION
server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc
webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 65a9aa328
webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64
webapp/src/test/resources/json/audit-search-parameter-purge.json PRE-CREATION
Diff: https://reviews.apache.org/r/71791/diff/16/
Changes: https://reviews.apache.org/r/71791/diff/15-16/
Testing
-------
Two UT and one IT has been added to test functionality like:
1. Atlas Audit entry save and retrive
2. Atlas Purge Audit store, retrive and search (UT and IT)
Thanks,
Sidharth Mishra