You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2018/06/13 22:31:00 UTC
Review Request 67586: Export: Support for Incremental Export
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67586/
-----------------------------------------------------------
Review request for atlas, Madhan Neethiraj and Ruchi Solani.
Bugs: ATLAS-2738
https://issues.apache.org/jira/browse/ATLAS-2738
Repository: atlas
Description
-------
**Approach**
Existing _ExportService_ has been extended.
_AtlasExportRequest_ supports: _fetchType_ as _incremental_. This takes additional parameter _fromTimestamp_ of type long.
During export process, the entities that are encountered added to _ZipSink_ only when their _modifiedTimestamp_ is less than the _fromTimestamp_ value.
This option can thus be applied with all the existing _matchType_.
**Updates to AtlasClient**
_AtlasClient_ now support _exportData_ API. This allows for _ExportService_ to be accessed via client APIs.
**Documentation**
None.
Diffs
-----
client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java aa1c77389
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java f1ed82210
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 8f3075e3d
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java ce792e0a2
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java 682c7bf0e
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceIncrementalTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 8b68bba24
repository/src/test/resources/json/stocksDB-Entities/db.json PRE-CREATION
repository/src/test/resources/json/stocksDB-Entities/export-incremental.json PRE-CREATION
repository/src/test/resources/json/stocksDB-Entities/table-columns.json PRE-CREATION
repository/src/test/resources/json/stocksDB-Entities/typesdef-new-classification.json PRE-CREATION
repository/src/test/resources/json/stocksDB-Entities/typesdef.json PRE-CREATION
stocks-base.zip PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/resources/AdminExportImportTestIT.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/resources/AdminExportTest.java a8f82321b
webapp/src/test/resources/json/export-incremental.json PRE-CREATION
webapp/src/test/resources/stocks-base.zip PRE-CREATION
Diff: https://reviews.apache.org/r/67586/diff/1/
Testing
-------
**Unit tests**
Tests that exercise the new option added.
**Integration tests**
New test added that perfom import and export with export option set to incremental.
File Attachments
----------------
Export Option set to Incremental
https://reviews.apache.org/media/uploaded/files/2018/06/13/e1822097-c386-4618-90c9-de5116dddebb__export-incremental.json
Thanks,
Ashutosh Mestry
Re: Review Request 67586: Export: Support for Incremental Export
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67586/
-----------------------------------------------------------
(Updated Aug. 6, 2018, 11:10 p.m.)
Review request for atlas, Madhan Neethiraj and Ruchi Solani.
Changes
-------
Updates include: Refactoring.
Bugs: ATLAS-2738
https://issues.apache.org/jira/browse/ATLAS-2738
Repository: atlas
Description
-------
**Approach**
Existing _ExportService_ has been extended.
_AtlasExportRequest_ supports: _fetchType_ as _incremental_. This takes additional parameter _fromTimestamp_ of type long.
During export process, the entities that are encountered added to _ZipSink_ only when their _modifiedTimestamp_ is less than the _fromTimestamp_ value.
This option can thus be applied with all the existing _matchType_.
**Updates to AtlasClient**
_AtlasClient_ now support _exportData_ API. This allows for _ExportService_ to be accessed via client APIs.
_AtlasClient_ has additional _importData_ API. This takes _byte_ array as parameter, thus allowing imports without needing absolute file name.
**Documentation**
None.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java 035216bc45e9c684bf9f49f13e5523570603d997
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 4da91a041593eaef8449c641263052f05802e914
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java eeb873511fcbf7392e9ebf7239cbb436165237cf
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 7a612090158130e53ded01a8248293c3e46ea43d
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 5ae86ef3b6f057ac08368d86c445c53fa0a64641
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java f465d678d7d603918ef8f74e93dea462b523b6e4
repository/src/test/resources/json/stocksDB-Entities/export-incremental.json PRE-CREATION
Diff: https://reviews.apache.org/r/67586/diff/4/
Changes: https://reviews.apache.org/r/67586/diff/3-4/
Testing
-------
**Unit tests**
Tests that exercise the new option added.
**Integration tests**
New test added that perfom import and export with export option set to incremental.
File Attachments
----------------
Export Option set to Incremental
https://reviews.apache.org/media/uploaded/files/2018/06/13/e1822097-c386-4618-90c9-de5116dddebb__export-incremental.json
Thanks,
Ashutosh Mestry
Re: Review Request 67586: Export: Support for Incremental Export
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67586/
-----------------------------------------------------------
(Updated June 13, 2018, 10:55 p.m.)
Review request for atlas, Madhan Neethiraj and Ruchi Solani.
Changes
-------
Updates include: Additional API for _importData_.
Bugs: ATLAS-2738
https://issues.apache.org/jira/browse/ATLAS-2738
Repository: atlas
Description (updated)
-------
**Approach**
Existing _ExportService_ has been extended.
_AtlasExportRequest_ supports: _fetchType_ as _incremental_. This takes additional parameter _fromTimestamp_ of type long.
During export process, the entities that are encountered added to _ZipSink_ only when their _modifiedTimestamp_ is less than the _fromTimestamp_ value.
This option can thus be applied with all the existing _matchType_.
**Updates to AtlasClient**
_AtlasClient_ now support _exportData_ API. This allows for _ExportService_ to be accessed via client APIs.
_AtlasClient_ has additional _importData_ API. This takes _byte_ array as parameter, thus allowing imports without needing absolute file name.
**Documentation**
None.
Diffs (updated)
-----
client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java aa1c77389
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java f1ed82210
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 8f3075e3d
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java ce792e0a2
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java 682c7bf0e
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceIncrementalTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 8b68bba24
repository/src/test/resources/json/stocksDB-Entities/db.json PRE-CREATION
repository/src/test/resources/json/stocksDB-Entities/export-incremental.json PRE-CREATION
repository/src/test/resources/json/stocksDB-Entities/table-columns.json PRE-CREATION
repository/src/test/resources/json/stocksDB-Entities/typesdef-new-classification.json PRE-CREATION
repository/src/test/resources/json/stocksDB-Entities/typesdef.json PRE-CREATION
stocks-base.zip PRE-CREATION
typesystem/src/test/resources/atlas-application.properties 65dd9a31f
webapp/src/test/java/org/apache/atlas/web/resources/AdminExportImportTestIT.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/resources/AdminExportTest.java a8f82321b
webapp/src/test/resources/ctas.zip PRE-CREATION
webapp/src/test/resources/json/export-incremental.json PRE-CREATION
webapp/src/test/resources/stocks-base.zip PRE-CREATION
Diff: https://reviews.apache.org/r/67586/diff/2/
Changes: https://reviews.apache.org/r/67586/diff/1-2/
Testing
-------
**Unit tests**
Tests that exercise the new option added.
**Integration tests**
New test added that perfom import and export with export option set to incremental.
File Attachments
----------------
Export Option set to Incremental
https://reviews.apache.org/media/uploaded/files/2018/06/13/e1822097-c386-4618-90c9-de5116dddebb__export-incremental.json
Thanks,
Ashutosh Mestry