You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Nikhil Bonte <ni...@freestoneinfotech.com> on 2021/06/15 13:15:10 UTC
Review Request 73423: ATLAS-4338: Import Service: Support change
notifiers for migration import
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/
-----------------------------------------------------------
Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.
Bugs: ATLAS-4338
https://issues.apache.org/jira/browse/ATLAS-4338
Repository: atlas
Description
-------
Currently Migration import feature (MigrationImport.java) does not support enabling notifiers (audits writer, notification to Atlas entities topic).
With this improvement, Atlas now will enable notifiers for Migration import, based on a flag.
Approach:
* New option in AtlasImportRequest - sendNotifications (default : false)
* MigrationImport will use the flag to select appropriate implementation of IAtlasEntityChangeNotifier to create EntityCreationManager.
* EntityCreationManager will then notify listeners.
Diffs
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 1d29bf833
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java 99be169f2
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java 8e17fd410
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java d6f23d6e2
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java b73988fd7
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java 734add6d7
Diff: https://reviews.apache.org/r/73423/diff/1/
Testing
-------
* Tested Manually
* Attached sample zip, sample request
{
"options": {
"migration": "true",
"size": "100",
"sendNotifications": "true",
"batchSize": 500,
"numWorkers": 40
}
}
* took ~270 secs to import 12987 entities with notifiers enabled.
* PC build :
File Attachments
----------------
largedb.zip
https://reviews.apache.org/media/uploaded/files/2021/06/15/c84f450b-c2b3-4cac-8b4a-76942a2c14e3__largedb.zip
Thanks,
Nikhil Bonte
Re: Review Request 73423: ATLAS-4338: Import Service: Support change
notifiers for migration import
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/#review223211
-----------------------------------------------------------
Ship it!
Ship It!
- Ashutosh Mestry
On June 30, 2021, 11:07 a.m., Nikhil Bonte wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73423/
> -----------------------------------------------------------
>
> (Updated June 30, 2021, 11:07 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4338
> https://issues.apache.org/jira/browse/ATLAS-4338
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Migration import feature (MigrationImport.java) does not support enabling notifiers (audits writer, notification to Atlas entities topic).
>
> With this improvement, Atlas now will enable notifiers for Migration import, based on a flag.
>
>
>
> Approach:
>
> * New option in AtlasImportRequest - sendNotifications (default : false)
> * MigrationImport will use the flag to select appropriate implementation of IAtlasEntityChangeNotifier to create EntityCreationManager.
> * EntityCreationManager will then notify listeners.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 1d29bf833
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java 99be169f2
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java 8e17fd410
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/GuidMutationResponsePair.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java d6f23d6e2
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java b73988fd7
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java 734add6d7
>
>
> Diff: https://reviews.apache.org/r/73423/diff/3/
>
>
> Testing
> -------
>
> * Tested Manually
> * Sample request
> {
> "options": {
> "migration": "true",
> "size": "100",
> "sendNotifications": "true",
> "batchSize": 500,
> "numWorkers": 40
> }
> }
> * took ~270 secs to import 12987 entities with notifiers enabled.
> * PC build : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console
>
>
> Thanks,
>
> Nikhil Bonte
>
>
Re: Review Request 73423: ATLAS-4338: Import Service: Support change
notifiers for migration import
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/
-----------------------------------------------------------
(Updated June 30, 2021, 4:37 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.
Changes
-------
Addressed review comments
Bugs: ATLAS-4338
https://issues.apache.org/jira/browse/ATLAS-4338
Repository: atlas
Description
-------
Currently Migration import feature (MigrationImport.java) does not support enabling notifiers (audits writer, notification to Atlas entities topic).
With this improvement, Atlas now will enable notifiers for Migration import, based on a flag.
Approach:
* New option in AtlasImportRequest - sendNotifications (default : false)
* MigrationImport will use the flag to select appropriate implementation of IAtlasEntityChangeNotifier to create EntityCreationManager.
* EntityCreationManager will then notify listeners.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 1d29bf833
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java 99be169f2
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java 8e17fd410
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/GuidMutationResponsePair.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java d6f23d6e2
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java b73988fd7
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java 734add6d7
Diff: https://reviews.apache.org/r/73423/diff/3/
Changes: https://reviews.apache.org/r/73423/diff/2-3/
Testing
-------
* Tested Manually
* Sample request
{
"options": {
"migration": "true",
"size": "100",
"sendNotifications": "true",
"batchSize": 500,
"numWorkers": 40
}
}
* took ~270 secs to import 12987 entities with notifiers enabled.
* PC build : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console
Thanks,
Nikhil Bonte
Re: Review Request 73423: ATLAS-4338: Import Service: Support change
notifiers for migration import
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/#review223196
-----------------------------------------------------------
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 58 (patched)
<https://reviews.apache.org/r/73423/#comment312296>
Rename: OPTION_MIGRATION_WITH_AUDITS
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 170 (patched)
<https://reviews.apache.org/r/73423/#comment312297>
Rename: getOptionMigrationWithAudis
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/BulkImportResultItem.java
Lines 23 (patched)
<https://reviews.apache.org/r/73423/#comment312298>
Rename: GuidResponsePair
- Ashutosh Mestry
On June 23, 2021, 2:21 p.m., Nikhil Bonte wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73423/
> -----------------------------------------------------------
>
> (Updated June 23, 2021, 2:21 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4338
> https://issues.apache.org/jira/browse/ATLAS-4338
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Migration import feature (MigrationImport.java) does not support enabling notifiers (audits writer, notification to Atlas entities topic).
>
> With this improvement, Atlas now will enable notifiers for Migration import, based on a flag.
>
>
>
> Approach:
>
> * New option in AtlasImportRequest - sendNotifications (default : false)
> * MigrationImport will use the flag to select appropriate implementation of IAtlasEntityChangeNotifier to create EntityCreationManager.
> * EntityCreationManager will then notify listeners.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 1d29bf833
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java 99be169f2
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java 8e17fd410
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/BulkImportResultItem.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java d6f23d6e2
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java b73988fd7
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java 734add6d7
>
>
> Diff: https://reviews.apache.org/r/73423/diff/2/
>
>
> Testing
> -------
>
> * Tested Manually
> * Sample request
> {
> "options": {
> "migration": "true",
> "size": "100",
> "sendNotifications": "true",
> "batchSize": 500,
> "numWorkers": 40
> }
> }
> * took ~270 secs to import 12987 entities with notifiers enabled.
> * PC build : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console
>
>
> Thanks,
>
> Nikhil Bonte
>
>
Re: Review Request 73423: ATLAS-4338: Import Service: Support change
notifiers for migration import
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/
-----------------------------------------------------------
(Updated June 23, 2021, 7:51 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.
Changes
-------
Addressed review comment
Bugs: ATLAS-4338
https://issues.apache.org/jira/browse/ATLAS-4338
Repository: atlas
Description
-------
Currently Migration import feature (MigrationImport.java) does not support enabling notifiers (audits writer, notification to Atlas entities topic).
With this improvement, Atlas now will enable notifiers for Migration import, based on a flag.
Approach:
* New option in AtlasImportRequest - sendNotifications (default : false)
* MigrationImport will use the flag to select appropriate implementation of IAtlasEntityChangeNotifier to create EntityCreationManager.
* EntityCreationManager will then notify listeners.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 1d29bf833
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java 99be169f2
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java 8e17fd410
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/BulkImportResultItem.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java d6f23d6e2
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java b73988fd7
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java 734add6d7
Diff: https://reviews.apache.org/r/73423/diff/2/
Changes: https://reviews.apache.org/r/73423/diff/1-2/
Testing
-------
* Tested Manually
* Sample request
{
"options": {
"migration": "true",
"size": "100",
"sendNotifications": "true",
"batchSize": 500,
"numWorkers": 40
}
}
* took ~270 secs to import 12987 entities with notifiers enabled.
* PC build : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console
Thanks,
Nikhil Bonte
Re: Review Request 73423: ATLAS-4338: Import Service: Support change
notifiers for migration import
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/#review223184
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
Lines 118 (patched)
<https://reviews.apache.org/r/73423/#comment312282>
Refactor: Move to separate file.
- Ashutosh Mestry
On June 21, 2021, 2:06 p.m., Nikhil Bonte wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73423/
> -----------------------------------------------------------
>
> (Updated June 21, 2021, 2:06 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4338
> https://issues.apache.org/jira/browse/ATLAS-4338
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Migration import feature (MigrationImport.java) does not support enabling notifiers (audits writer, notification to Atlas entities topic).
>
> With this improvement, Atlas now will enable notifiers for Migration import, based on a flag.
>
>
>
> Approach:
>
> * New option in AtlasImportRequest - sendNotifications (default : false)
> * MigrationImport will use the flag to select appropriate implementation of IAtlasEntityChangeNotifier to create EntityCreationManager.
> * EntityCreationManager will then notify listeners.
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 1d29bf833
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java 99be169f2
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java 8e17fd410
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java d6f23d6e2
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java b73988fd7
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java 734add6d7
>
>
> Diff: https://reviews.apache.org/r/73423/diff/1/
>
>
> Testing
> -------
>
> * Tested Manually
> * Sample request
> {
> "options": {
> "migration": "true",
> "size": "100",
> "sendNotifications": "true",
> "batchSize": 500,
> "numWorkers": 40
> }
> }
> * took ~270 secs to import 12987 entities with notifiers enabled.
> * PC build : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console
>
>
> Thanks,
>
> Nikhil Bonte
>
>
Re: Review Request 73423: ATLAS-4338: Import Service: Support change
notifiers for migration import
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/
-----------------------------------------------------------
(Updated June 21, 2021, 7:36 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.
Changes
-------
Updated PC build link
Bugs: ATLAS-4338
https://issues.apache.org/jira/browse/ATLAS-4338
Repository: atlas
Description
-------
Currently Migration import feature (MigrationImport.java) does not support enabling notifiers (audits writer, notification to Atlas entities topic).
With this improvement, Atlas now will enable notifiers for Migration import, based on a flag.
Approach:
* New option in AtlasImportRequest - sendNotifications (default : false)
* MigrationImport will use the flag to select appropriate implementation of IAtlasEntityChangeNotifier to create EntityCreationManager.
* EntityCreationManager will then notify listeners.
Diffs
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 1d29bf833
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java 99be169f2
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java 8e17fd410
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java d6f23d6e2
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java b73988fd7
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java 734add6d7
Diff: https://reviews.apache.org/r/73423/diff/1/
Testing (updated)
-------
* Tested Manually
* Sample request
{
"options": {
"migration": "true",
"size": "100",
"sendNotifications": "true",
"batchSize": 500,
"numWorkers": 40
}
}
* took ~270 secs to import 12987 entities with notifiers enabled.
* PC build : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console
Thanks,
Nikhil Bonte