You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2018/08/06 23:10:04 UTC
Re: Review Request 67586: Export: Support for Incremental Export

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67586/
-----------------------------------------------------------

(Updated Aug. 6, 2018, 11:10 p.m.)


Review request for atlas, Madhan Neethiraj and Ruchi Solani.


Changes
-------

Updates include: Refactoring.


Bugs: ATLAS-2738
    https://issues.apache.org/jira/browse/ATLAS-2738


Repository: atlas


Description
-------

**Approach**
Existing _ExportService_ has been extended.

_AtlasExportRequest_ supports: _fetchType_ as _incremental_. This takes additional parameter _fromTimestamp_ of type long. 

During export process, the entities that are encountered added to _ZipSink_ only when their _modifiedTimestamp_ is less than the _fromTimestamp_ value. 

This option can thus be applied with all the existing _matchType_.

**Updates to AtlasClient**
_AtlasClient_ now support _exportData_ API. This allows for _ExportService_ to be accessed via client APIs.
_AtlasClient_ has additional _importData_ API. This takes _byte_ array as parameter, thus allowing imports without needing absolute file name.

**Documentation**
None.


Diffs (updated)
-----

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java 035216bc45e9c684bf9f49f13e5523570603d997 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 4da91a041593eaef8449c641263052f05802e914 
  repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java eeb873511fcbf7392e9ebf7239cbb436165237cf 
  repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 7a612090158130e53ded01a8248293c3e46ea43d 
  repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 5ae86ef3b6f057ac08368d86c445c53fa0a64641 
  repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java f465d678d7d603918ef8f74e93dea462b523b6e4 
  repository/src/test/resources/json/stocksDB-Entities/export-incremental.json PRE-CREATION 


Diff: https://reviews.apache.org/r/67586/diff/4/

Changes: https://reviews.apache.org/r/67586/diff/3-4/


Testing
-------

**Unit tests**
Tests that exercise the new option added.

**Integration tests**
New test added that perfom import and export with export option set to incremental.


File Attachments
----------------

Export Option set to Incremental
  https://reviews.apache.org/media/uploaded/files/2018/06/13/e1822097-c386-4618-90c9-de5116dddebb__export-incremental.json


Thanks,

Ashutosh Mestry