You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Madhan Neethiraj <ma...@apache.org> on 2019/12/01 16:37:29 UTC
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for
Atlas Purge Entity
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218865
-----------------------------------------------------------
addons/models/0000-Area0/0010-base_model.json
Lines 4 (patched)
<https://reviews.apache.org/r/71791/#comment306755>
atlas_operation => atlas_admin_operation
addons/models/0000-Area0/0010-base_model.json
Lines 5 (patched)
<https://reviews.apache.org/r/71791/#comment306756>
"Audit Operation defines the operations we support for admin audit" => "Defines admin operations in Apache Atlas"
addons/models/0000-Area0/0010-base_model.json
Lines 426 (patched)
<https://reviews.apache.org/r/71791/#comment306762>
operationStartTime => startTime
addons/models/0000-Area0/0010-base_model.json
Lines 427 (patched)
<https://reviews.apache.org/r/71791/#comment306757>
long => Date
addons/models/0000-Area0/0010-base_model.json
Lines 434 (patched)
<https://reviews.apache.org/r/71791/#comment306763>
operationEndTime => endTime
addons/models/0000-Area0/0010-base_model.json
Lines 435 (patched)
<https://reviews.apache.org/r/71791/#comment306758>
long => Date
addons/models/0000-Area0/0010-base_model.json
Lines 450 (patched)
<https://reviews.apache.org/r/71791/#comment306764>
operationParams => params
addons/models/0000-Area0/0010-base_model.json
Lines 458 (patched)
<https://reviews.apache.org/r/71791/#comment306765>
operationResult => result
intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java
Lines 47 (patched)
<https://reviews.apache.org/r/71791/#comment306759>
long => Date - for both startTime and endTime
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
Lines 37 (patched)
<https://reviews.apache.org/r/71791/#comment306761>
PROPERTY_ => ATTRIBUTE_
repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
Lines 42 (patched)
<https://reviews.apache.org/r/71791/#comment306760>
CLIENT_HOST => CLIENT_ID
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 436 (patched)
<https://reviews.apache.org/r/71791/#comment306766>
Instead of generating audit in a seperate graph transaction here, consider generating in the transaction used by AtlasEntityStore.purgeByIds() i.e. move auditing to inside AtlasEntityStore.purgeByIds().
- Madhan Neethiraj
On Nov. 30, 2019, 7:44 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> -----------------------------------------------------------
>
> (Updated Nov. 30, 2019, 7:44 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Right now for purge, delete entity, etc. at Atlas we create audit entry at HBase. User can go to entity and click on audit to see the audit information for that particular entity. But if user purge one entity there will not be any vertex at Janus graph and user will not able to get the purge audit which is important for governance purpose.
>
> As part of fix, implemented Atlas Audit Framework which is being used for Purge operation and can be extended for other operations. In Future once we change the AtlasAudit type as super type for Import export as well we can use this framework for all Atlas operations.
>
> admin/audit rest has been added which will return all the Atlas audit operations including import export at Atlas.
>
>
> Diffs
> -----
>
> addons/models/0000-Area0/0010-base_model.json 2f5fdaf14
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c
> dashboardv2/public/js/utils/Enums.js 58542ec59
> dashboardv3/public/js/utils/Enums.js 58542ec59
> intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java b448d510b
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java 621458406
> repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187
> repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java be1698fb4
> repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java 78865c569
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java ba7a8a04b
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 925b2a007
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 4d438525a
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java db8c6c83c
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 25e0a5362
> repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java 1863b8d74
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java d0c06a107
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 78fdaca8d
> repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java 42b63535b
> repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java a1b512fb9
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 7a1ed18cb
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java 48b958271
> repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java c9bb11c56
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 0ffc3d595
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 18e950e8c
> repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java 050bda3ae
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java 6f9c05e7a
> repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java e56e48308
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java baeafd4b7
> repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc
> webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49
> webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64
>
>
> Diff: https://reviews.apache.org/r/71791/diff/5/
>
>
> Testing
> -------
>
> Two UT and one IT has been added to test functionality like:
> 1. Atlas Audit entry save and retrive
> 2. Atlas Purge Audit store, retrive and search (UT and IT)
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for
Atlas Purge Entity
Posted by Sidharth Mishra <si...@gmail.com>.
> On Dec. 1, 2019, 4:37 p.m., Madhan Neethiraj wrote:
> > addons/models/0000-Area0/0010-base_model.json
> > Lines 5 (patched)
> > <https://reviews.apache.org/r/71791/diff/5/?file=2181198#file2181198line5>
> >
> > "Audit Operation defines the operations we support for admin audit" => "Defines admin operations in Apache Atlas"
Removed the admin and changed it to "Defines operations in Apache Atlas which can be audited"
> On Dec. 1, 2019, 4:37 p.m., Madhan Neethiraj wrote:
> > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> > Lines 436 (patched)
> > <https://reviews.apache.org/r/71791/diff/5/?file=2181235#file2181235line461>
> >
> > Instead of generating audit in a seperate graph transaction here, consider generating in the transaction used by AtlasEntityStore.purgeByIds() i.e. move auditing to inside AtlasEntityStore.purgeByIds().
Moving auditing to inside AtlasEntityStore.purgeByIds() is causing circular Injection issue. AtlasAuditService depends on DataAccess and DataAccess depends on AtlasEntityStore. So AtlasEntityStore can't depends on AtlasAuditService again.
- Sidharth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218865
-----------------------------------------------------------
On Dec. 4, 2019, 9:47 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> -----------------------------------------------------------
>
> (Updated Dec. 4, 2019, 9:47 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Right now for purge, delete entity, etc. at Atlas we create audit entry at HBase. User can go to entity and click on audit to see the audit information for that particular entity. But if user purge one entity there will not be any vertex at Janus graph and user will not able to get the purge audit which is important for governance purpose.
>
> As part of fix, implemented Atlas Audit Framework which is being used for Purge operation and can be extended for other operations. In Future once we change the AtlasAudit type as super type for Import export as well we can use this framework for all Atlas operations.
>
> admin/audit rest has been added which will return all the Atlas audit operations including import export at Atlas.
>
>
> Diffs
> -----
>
> addons/models/0000-Area0/0010-base_model.json 2f5fdaf14
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c
> dashboardv2/public/js/utils/Enums.js 58542ec59
> dashboardv3/public/js/utils/Enums.js 58542ec59
> intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java b448d510b
> intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java eb8634533
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java c9ed79750
> repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187
> repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java be1698fb4
> repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java 78865c569
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java ba7a8a04b
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 925b2a007
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 4d438525a
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java db8c6c83c
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 25e0a5362
> repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java 1863b8d74
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java d0c06a107
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 78fdaca8d
> repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java 42b63535b
> repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java a1b512fb9
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 7a1ed18cb
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java 48b958271
> repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java c9bb11c56
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 0ffc3d595
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 18e950e8c
> repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java 050bda3ae
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java 6f9c05e7a
> repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java e56e48308
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java baeafd4b7
> repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc
> webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49
> webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64
>
>
> Diff: https://reviews.apache.org/r/71791/diff/6/
>
>
> Testing
> -------
>
> Two UT and one IT has been added to test functionality like:
> 1. Atlas Audit entry save and retrive
> 2. Atlas Purge Audit store, retrive and search (UT and IT)
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for
Atlas Purge Entity
Posted by Sidharth Mishra <si...@gmail.com>.
> On Dec. 1, 2019, 4:37 p.m., Madhan Neethiraj wrote:
> > addons/models/0000-Area0/0010-base_model.json
> > Lines 4 (patched)
> > <https://reviews.apache.org/r/71791/diff/5/?file=2181198#file2181198line4>
> >
> > atlas_operation => atlas_admin_operation
I think as you initially suggested to rename it to atlas_operation is more flexible than atlas_admin_operation as in future we may allow some more operation to be audited like other than admin operations.
- Sidharth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218865
-----------------------------------------------------------
On Nov. 30, 2019, 7:44 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> -----------------------------------------------------------
>
> (Updated Nov. 30, 2019, 7:44 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Right now for purge, delete entity, etc. at Atlas we create audit entry at HBase. User can go to entity and click on audit to see the audit information for that particular entity. But if user purge one entity there will not be any vertex at Janus graph and user will not able to get the purge audit which is important for governance purpose.
>
> As part of fix, implemented Atlas Audit Framework which is being used for Purge operation and can be extended for other operations. In Future once we change the AtlasAudit type as super type for Import export as well we can use this framework for all Atlas operations.
>
> admin/audit rest has been added which will return all the Atlas audit operations including import export at Atlas.
>
>
> Diffs
> -----
>
> addons/models/0000-Area0/0010-base_model.json 2f5fdaf14
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c
> dashboardv2/public/js/utils/Enums.js 58542ec59
> dashboardv3/public/js/utils/Enums.js 58542ec59
> intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java b448d510b
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java 621458406
> repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187
> repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java be1698fb4
> repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java 78865c569
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java ba7a8a04b
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 925b2a007
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 4d438525a
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java db8c6c83c
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 25e0a5362
> repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java 1863b8d74
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java d0c06a107
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 78fdaca8d
> repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java 42b63535b
> repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java a1b512fb9
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 7a1ed18cb
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java 48b958271
> repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java c9bb11c56
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 0ffc3d595
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 18e950e8c
> repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java 050bda3ae
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java 6f9c05e7a
> repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java e56e48308
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java baeafd4b7
> repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/RequestContext.java f9ca7a26d
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 40fd6efdc
> webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java a4b7b0d49
> webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 563a16f64
>
>
> Diff: https://reviews.apache.org/r/71791/diff/5/
>
>
> Testing
> -------
>
> Two UT and one IT has been added to test functionality like:
> 1. Atlas Audit entry save and retrive
> 2. Atlas Purge Audit store, retrive and search (UT and IT)
>
>
> Thanks,
>
> Sidharth Mishra
>
>