You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Nikhil Bonte <ni...@freestoneinfotech.com> on 2020/02/13 13:02:00 UTC
Re: Review Request 72039: ATLAS-3595,
ATLAS-3603 Migration import: support zip file for data migration
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72039/
-----------------------------------------------------------
(Updated Feb. 13, 2020, 1:02 p.m.)
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
* Addressed review comments.
* Included ATLAS-3603.
Summary (updated)
-----------------
ATLAS-3595, ATLAS-3603 Migration import: support zip file for data migration
Bugs: ATLAS-3595
https://issues.apache.org/jira/browse/ATLAS-3595
Repository: atlas
Description (updated)
-------
## ATLAS-3595
Data migration service added under ATLAS-2461 works with json files as input.
This improvement will enable Data Migration service to import data using a zip file.
Approach:
* Property value for migration should contain full path of file, below is the example.
* Data migration service will first check migration property if value refers to a zip file.
* If true, it will call ZipMigrationImporter to import data in zip file.
* ZipMigrationImporter internally uses existing ImportService to ingest zip.
* Audit service & Notification service will also be started as a part of zip import.
E.g. properties to enable zip data migration:
* atlas.migration.data.filename=/path/fileName.zip
* atlas.migration.mode.workers=5
* atlas.migration.mode.batch.size=20
## ATLAS-3603
Migration import should start when the data (zip) to be ingested is available at the pre-configured location.
Approach:
ZipMigrationImporter will first check if zip is available, accessible & completely written.
If yes, migration will start ingesting zip.
if not, will poll the location periodically. when existence is detected, import will proceed as usual.
It uses the file size comparison approach. Once modify event is received, file size is compared in an interval. If sizes match, it means file is available and not being written.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/service/Services.java 7b36db5ff
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 170136195
repository/src/main/java/org/apache/atlas/repository/migration/FileWatcher.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/ZipMigrationImporter.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/MigrationImportTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/72039/diff/7/
Changes: https://reviews.apache.org/r/72039/diff/6-7/
Testing
-------
PC build - https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1614/
Thanks,
Nikhil Bonte
Re: Review Request 72039: ATLAS-3595,
ATLAS-3603 Migration import: support zip file for data migration
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72039/#review219586
-----------------------------------------------------------
Ship it!
Ship It!
- Ashutosh Mestry
On Feb. 13, 2020, 1:02 p.m., Nikhil Bonte wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72039/
> -----------------------------------------------------------
>
> (Updated Feb. 13, 2020, 1:02 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3595
> https://issues.apache.org/jira/browse/ATLAS-3595
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ## ATLAS-3595
> Data migration service added under ATLAS-2461 works with json files as input.
>
> This improvement will enable Data Migration service to import data using a zip file.
>
> Approach:
>
> * Property value for migration should contain full path of file, below is the example.
> * Data migration service will first check migration property if value refers to a zip file.
> * If true, it will call ZipMigrationImporter to import data in zip file.
> * ZipMigrationImporter internally uses existing ImportService to ingest zip.
> * Audit service & Notification service will also be started as a part of zip import.
>
>
> E.g. properties to enable zip data migration:
>
> * atlas.migration.data.filename=/path/fileName.zip
> * atlas.migration.mode.workers=5
> * atlas.migration.mode.batch.size=20
>
>
> ## ATLAS-3603
> Migration import should start when the data (zip) to be ingested is available at the pre-configured location.
>
> Approach:
>
> ZipMigrationImporter will first check if zip is available, accessible & completely written.
>
> If yes, migration will start ingesting zip.
> if not, will poll the location periodically. when existence is detected, import will proceed as usual.
>
> It uses the file size comparison approach. Once modify event is received, file size is compared in an interval. If sizes match, it means file is available and not being written.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/service/Services.java 7b36db5ff
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 170136195
> repository/src/main/java/org/apache/atlas/repository/migration/FileWatcher.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/migration/ZipMigrationImporter.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/72039/diff/8/
>
>
> Testing
> -------
>
> PC build - https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1632/
>
>
> Thanks,
>
> Nikhil Bonte
>
>