You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry via Review Board <no...@reviews.apache.org> on 2019/09/03 17:04:24 UTC
Review Request 71426: Import Service: Support for Backing Directory
for Large Imports
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/
-----------------------------------------------------------
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Bugs: ATLAS-3396
https://issues.apache.org/jira/browse/ATLAS-3396
Repository: atlas
Description
-------
**Background**
The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
**Approach**
This is a backward compatible.
- - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
- Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
- Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
**Documentation**
Updated.
Diffs
-----
docs/src/site/twiki/Import-API-Options.twiki 4004e7013
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 6a9985642
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
Diff: https://reviews.apache.org/r/71426/diff/1/
Testing
-------
**Unit tests**
Updated to exercise the new implementation.
**Volume tests**
Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
**Functional tests**
Accuracy tests preformed on small size data.
Thanks,
Ashutosh Mestry
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/#review217604
-----------------------------------------------------------
docs/src/site/twiki/Import-API-Options.twiki
Lines 142 (patched)
<https://reviews.apache.org/r/71426/#comment304874>
Suggestion: Adding this property in "atlas-application.properties" template file as commented line can be helpful.
- Nikhil Bonte
On Sept. 5, 2019, 1:14 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71426/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2019, 1:14 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3396
> https://issues.apache.org/jira/browse/ATLAS-3396
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
>
> **Approach**
> This is a backward compatible.
>
> - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
> - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
> - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
>
>
> **CURL**
>
> Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
>
> _CURL_
> ```
> curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
> ```
>
> **Documentation**
> Updated.
>
>
> Diffs
> -----
>
> docs/src/site/twiki/Import-API-Options.twiki 4004e7013
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 6a9985642
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
>
>
> Diff: https://reviews.apache.org/r/71426/diff/4/
>
>
> Testing
> -------
>
> **Unit tests**
> Updated to exercise the new implementation.
>
> **Volume tests**
> Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
>
> **Functional tests**
> Accuracy tests preformed on small size data.
>
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/#review217664
-----------------------------------------------------------
distro/src/conf/atlas-application.properties
Lines 69 (patched)
<https://reviews.apache.org/r/71426/#comment304958>
atlas.import.temporaryDirectory => atlas.import.temp.directory
- Madhan Neethiraj
On Sept. 9, 2019, 7:16 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71426/
> -----------------------------------------------------------
>
> (Updated Sept. 9, 2019, 7:16 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3396
> https://issues.apache.org/jira/browse/ATLAS-3396
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
>
> **Approach**
> This is a backward compatible.
>
> - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
> - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
> - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
>
>
> **CURL**
>
> Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
>
> _CURL_
> ```
> curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
> ```
>
> **Documentation**
> Updated.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/AtlasConfiguration.java c5357f5a4
> distro/src/conf/atlas-application.properties 654a9304e
> docs/src/site/twiki/Import-API-Options.twiki 4004e7013
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 016acd743
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java 26066920a
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java a11bcdcb4
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 5ec62e5f9
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 06bdaa6d6
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 5e909e198
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
>
>
> Diff: https://reviews.apache.org/r/71426/diff/8/
>
>
> Testing
> -------
>
> **Unit tests**
> Updated to exercise the new implementation.
>
> **Volume tests**
> Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
>
> **Functional tests**
> Accuracy tests preformed on small size data.
>
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/
-----------------------------------------------------------
(Updated Sept. 9, 2019, 10:54 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include: Addressed review comments.
Bugs: ATLAS-3396
https://issues.apache.org/jira/browse/ATLAS-3396
Repository: atlas
Description
-------
**Background**
The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
**Approach**
This is a backward compatible.
- - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
- Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
- Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
**CURL**
Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
_CURL_
```
curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
```
**Documentation**
Updated.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConfiguration.java c5357f5a4
distro/src/conf/atlas-application.properties 654a9304e
docs/src/site/twiki/Import-API-Options.twiki 4004e7013
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 016acd743
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java 26066920a
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java a11bcdcb4
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 5ec62e5f9
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 06bdaa6d6
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 5e909e198
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 7436dc0c5
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
Diff: https://reviews.apache.org/r/71426/diff/9/
Changes: https://reviews.apache.org/r/71426/diff/8-9/
Testing
-------
**Unit tests**
Updated to exercise the new implementation.
**Volume tests**
Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
**Functional tests**
Accuracy tests preformed on small size data.
**Pre-commit Build**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
Thanks,
Ashutosh Mestry
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/
-----------------------------------------------------------
(Updated Sept. 9, 2019, 7:16 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include: Rebased.
Bugs: ATLAS-3396
https://issues.apache.org/jira/browse/ATLAS-3396
Repository: atlas
Description
-------
**Background**
The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
**Approach**
This is a backward compatible.
- - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
- Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
- Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
**CURL**
Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
_CURL_
```
curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
```
**Documentation**
Updated.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConfiguration.java c5357f5a4
distro/src/conf/atlas-application.properties 654a9304e
docs/src/site/twiki/Import-API-Options.twiki 4004e7013
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 016acd743
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java 26066920a
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java a11bcdcb4
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 5ec62e5f9
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 06bdaa6d6
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 5e909e198
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
Diff: https://reviews.apache.org/r/71426/diff/8/
Changes: https://reviews.apache.org/r/71426/diff/7-8/
Testing
-------
**Unit tests**
Updated to exercise the new implementation.
**Volume tests**
Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
**Functional tests**
Accuracy tests preformed on small size data.
**Pre-commit Build**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
Thanks,
Ashutosh Mestry
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/
-----------------------------------------------------------
(Updated Sept. 9, 2019, 6:23 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include: Addressed review comments.
Bugs: ATLAS-3396
https://issues.apache.org/jira/browse/ATLAS-3396
Repository: atlas
Description
-------
**Background**
The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
**Approach**
This is a backward compatible.
- - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
- Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
- Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
**CURL**
Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
_CURL_
```
curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
```
**Documentation**
Updated.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConfiguration.java c5357f5a4
distro/src/conf/atlas-application.properties 654a9304e
docs/src/site/twiki/Import-API-Options.twiki 4004e7013
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 016acd743
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java 26066920a
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 06bdaa6d6
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
Diff: https://reviews.apache.org/r/71426/diff/7/
Changes: https://reviews.apache.org/r/71426/diff/6-7/
Testing
-------
**Unit tests**
Updated to exercise the new implementation.
**Volume tests**
Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
**Functional tests**
Accuracy tests preformed on small size data.
**Pre-commit Build**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
Thanks,
Ashutosh Mestry
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/#review217657
-----------------------------------------------------------
Fix it, then Ship it!
distro/src/conf/atlas-application.properties
Lines 69 (patched)
<https://reviews.apache.org/r/71426/#comment304949>
atlas.import.temporaryDirectory => atlas.import.temp.directory
docs/src/site/twiki/Import-API-Options.twiki
Lines 144 (patched)
<https://reviews.apache.org/r/71426/#comment304950>
"It is up to the user to ascertain" => "Please ensure "
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java
Line 99 (original), 103 (patched)
<https://reviews.apache.org/r/71426/#comment304943>
'@Override' seems to be missing for few methods - like getTypesDef(), getExportResult(). Please review and update.
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java
Lines 74 (patched)
<https://reviews.apache.org/r/71426/#comment304951>
please move 'private' methods after all public and protected methods.
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java
Lines 52 (patched)
<https://reviews.apache.org/r/71426/#comment304948>
Please avoid statics *as much as possible*. Replace use of directoryBasedImportConfigured with:
if (StringUtils.isEmpty(AtlasConfiguration.IMPORT_TEMP_DIRECTORY.getString())) {
}
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java
Lines 220 (patched)
<https://reviews.apache.org/r/71426/#comment304947>
Consider replacing this block, #220 - #231, with a following:
package org.apache.atlas;
public enum AtlasConfiguration {
...
IMPORT_TEMP_DIRECTORY("atlas.import.temp.directory", "")
}
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
Line 102 (original), 103 (patched)
<https://reviews.apache.org/r/71426/#comment304946>
getInputStreamFrom(): this variation does return a ZipSource; so consider retaining current name, getZipSourceFrom().
- Madhan Neethiraj
On Sept. 9, 2019, 5:35 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71426/
> -----------------------------------------------------------
>
> (Updated Sept. 9, 2019, 5:35 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3396
> https://issues.apache.org/jira/browse/ATLAS-3396
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
>
> **Approach**
> This is a backward compatible.
>
> - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
> - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
> - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
>
>
> **CURL**
>
> Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
>
> _CURL_
> ```
> curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
> ```
>
> **Documentation**
> Updated.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 654a9304e
> docs/src/site/twiki/Import-API-Options.twiki 4004e7013
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 016acd743
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java 26066920a
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 06bdaa6d6
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
>
>
> Diff: https://reviews.apache.org/r/71426/diff/6/
>
>
> Testing
> -------
>
> **Unit tests**
> Updated to exercise the new implementation.
>
> **Volume tests**
> Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
>
> **Functional tests**
> Accuracy tests preformed on small size data.
>
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/
-----------------------------------------------------------
(Updated Sept. 9, 2019, 5:35 a.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include:
- Addressed review comments.
- Fixed performance problem that was causing slow ingest for entities with large lineage.
- Refactoring for reducing memory usage.
Bugs: ATLAS-3396
https://issues.apache.org/jira/browse/ATLAS-3396
Repository: atlas
Description
-------
**Background**
The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
**Approach**
This is a backward compatible.
- - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
- Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
- Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
**CURL**
Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
_CURL_
```
curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
```
**Documentation**
Updated.
Diffs (updated)
-----
distro/src/conf/atlas-application.properties 654a9304e
docs/src/site/twiki/Import-API-Options.twiki 4004e7013
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 016acd743
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java 26066920a
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 06bdaa6d6
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
Diff: https://reviews.apache.org/r/71426/diff/6/
Changes: https://reviews.apache.org/r/71426/diff/5-6/
Testing
-------
**Unit tests**
Updated to exercise the new implementation.
**Volume tests**
Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
**Functional tests**
Accuracy tests preformed on small size data.
**Pre-commit Build**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
Thanks,
Ashutosh Mestry
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/#review217625
-----------------------------------------------------------
Ship it!
Ship It!
- Sarath Subramanian
On Sept. 6, 2019, 2:09 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71426/
> -----------------------------------------------------------
>
> (Updated Sept. 6, 2019, 2:09 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3396
> https://issues.apache.org/jira/browse/ATLAS-3396
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
>
> **Approach**
> This is a backward compatible.
>
> - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
> - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
> - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
>
>
> **CURL**
>
> Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
>
> _CURL_
> ```
> curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
> ```
>
> **Documentation**
> Updated.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 654a9304e
> docs/src/site/twiki/Import-API-Options.twiki 4004e7013
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
>
>
> Diff: https://reviews.apache.org/r/71426/diff/5/
>
>
> Testing
> -------
>
> **Unit tests**
> Updated to exercise the new implementation.
>
> **Volume tests**
> Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
>
> **Functional tests**
> Accuracy tests preformed on small size data.
>
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
> On Sept. 7, 2019, 1:32 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java
> > Lines 73 (patched)
> > <https://reviews.apache.org/r/71426/diff/5/?file=2164217#file2164217line73>
> >
> > - updateGuidZipEntryMap() => initGuidZipEntryMap()
> > - consider moving #74 - #76 inside initGuidZipEntryMap()
Since I have changed the implementation.
> On Sept. 7, 2019, 1:32 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java
> > Lines 102 (patched)
> > <https://reviews.apache.org/r/71426/diff/5/?file=2164217#file2164217line102>
> >
> > Consider adding instance members for AtlasTypesDef and AtlasExportResult, creationOrder; and have them initialized in the constructor (or initGuidZipEntryMap()).
I have done an on-demand fetch for these.
> On Sept. 7, 2019, 1:32 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java
> > Lines 118 (patched)
> > <https://reviews.apache.org/r/71426/diff/5/?file=2164217#file2164217line118>
> >
> > permissionChecks() is called from only from here; it will be easier to read if this call is replaced with its implementation:
> > f.exists() && f.isDirectory() && f.canWrite()
Dropping since permissionChecks is more explanatory.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/#review217628
-----------------------------------------------------------
On Sept. 6, 2019, 9:09 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71426/
> -----------------------------------------------------------
>
> (Updated Sept. 6, 2019, 9:09 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3396
> https://issues.apache.org/jira/browse/ATLAS-3396
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
>
> **Approach**
> This is a backward compatible.
>
> - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
> - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
> - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
>
>
> **CURL**
>
> Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
>
> _CURL_
> ```
> curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
> ```
>
> **Documentation**
> Updated.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 654a9304e
> docs/src/site/twiki/Import-API-Options.twiki 4004e7013
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
>
>
> Diff: https://reviews.apache.org/r/71426/diff/5/
>
>
> Testing
> -------
>
> **Unit tests**
> Updated to exercise the new implementation.
>
> **Volume tests**
> Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
>
> **Functional tests**
> Accuracy tests preformed on small size data.
>
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/#review217628
-----------------------------------------------------------
For easier readability, consider arranging methods/members in the following order:
- static members: public, protected, private; within each accessor, ‘final’ members precede non-final
- instance members: public, protected, private; within each accessor, ‘final’ members precede non-final
- methods: public, protected, private; within each accessor, static methods precede non-static
- methods: all constructors immediately after ‘public static’ methods
- methods: all getters/setters immediately after constructors
- methods: all @Override methods immediately after getter/setters
docs/src/site/twiki/Import-API-Options.twiki
Lines 142 (patched)
<https://reviews.apache.org/r/71426/#comment304912>
Please review and update the comment, per the current implementation.
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java
Lines 57 (patched)
<https://reviews.apache.org/r/71426/#comment304919>
Looking at the implementation, guidEntityJsonMap doesn't seem to have any entry for entity at all. It only has entries for typedef/creation-order/export-result. Consider getting rid of 'guidEntityJsonMap' and have members for above 3 entries.
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java
Lines 73 (patched)
<https://reviews.apache.org/r/71426/#comment304915>
- updateGuidZipEntryMap() => initGuidZipEntryMap()
- consider moving #74 - #76 inside initGuidZipEntryMap()
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java
Lines 93 (patched)
<https://reviews.apache.org/r/71426/#comment304916>
add '@Override' to all methods that override base-class/interface.
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java
Lines 102 (patched)
<https://reviews.apache.org/r/71426/#comment304917>
Consider adding instance members for AtlasTypesDef and AtlasExportResult, creationOrder; and have them initialized in the constructor (or initGuidZipEntryMap()).
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java
Lines 118 (patched)
<https://reviews.apache.org/r/71426/#comment304918>
permissionChecks() is called from only from here; it will be easier to read if this call is replaced with its implementation:
f.exists() && f.isDirectory() && f.canWrite()
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
Line 96 (original), 99 (patched)
<https://reviews.apache.org/r/71426/#comment304914>
getZipSourceFrom() => getInputStreamFrom()
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Line 395 (original), 394 (patched)
<https://reviews.apache.org/r/71426/#comment304913>
'ZipSource' import, #45, is no more needed. Please review and remove.
- Madhan Neethiraj
On Sept. 6, 2019, 9:09 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71426/
> -----------------------------------------------------------
>
> (Updated Sept. 6, 2019, 9:09 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3396
> https://issues.apache.org/jira/browse/ATLAS-3396
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
>
> **Approach**
> This is a backward compatible.
>
> - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
> - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
> - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
>
>
> **CURL**
>
> Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
>
> _CURL_
> ```
> curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
> ```
>
> **Documentation**
> Updated.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 654a9304e
> docs/src/site/twiki/Import-API-Options.twiki 4004e7013
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
>
>
> Diff: https://reviews.apache.org/r/71426/diff/5/
>
>
> Testing
> -------
>
> **Unit tests**
> Updated to exercise the new implementation.
>
> **Volume tests**
> Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
>
> **Functional tests**
> Accuracy tests preformed on small size data.
>
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/
-----------------------------------------------------------
(Updated Sept. 6, 2019, 9:09 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include:
- Addressed review comments.
- Removed changes to _AtlasImportRequest_.
Bugs: ATLAS-3396
https://issues.apache.org/jira/browse/ATLAS-3396
Repository: atlas
Description
-------
**Background**
The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
**Approach**
This is a backward compatible.
- - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
- Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
- Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
**CURL**
Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
_CURL_
```
curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
```
**Documentation**
Updated.
Diffs (updated)
-----
distro/src/conf/atlas-application.properties 654a9304e
docs/src/site/twiki/Import-API-Options.twiki 4004e7013
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
Diff: https://reviews.apache.org/r/71426/diff/5/
Changes: https://reviews.apache.org/r/71426/diff/4-5/
Testing
-------
**Unit tests**
Updated to exercise the new implementation.
**Volume tests**
Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
**Functional tests**
Accuracy tests preformed on small size data.
**Pre-commit Build**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
Thanks,
Ashutosh Mestry
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/#review217618
-----------------------------------------------------------
docs/src/site/twiki/Import-API-Options.twiki
Lines 143 (patched)
<https://reviews.apache.org/r/71426/#comment304883>
> If this property is left blank, default temporary directory is used.
when this configuration is not set, Atlas should continue the current behavior - of using in-memory.
- Madhan Neethiraj
On Sept. 5, 2019, 1:14 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71426/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2019, 1:14 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3396
> https://issues.apache.org/jira/browse/ATLAS-3396
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
>
> **Approach**
> This is a backward compatible.
>
> - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
> - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
> - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
>
>
> **CURL**
>
> Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
>
> _CURL_
> ```
> curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
> ```
>
> **Documentation**
> Updated.
>
>
> Diffs
> -----
>
> docs/src/site/twiki/Import-API-Options.twiki 4004e7013
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 6a9985642
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
>
>
> Diff: https://reviews.apache.org/r/71426/diff/4/
>
>
> Testing
> -------
>
> **Unit tests**
> Updated to exercise the new implementation.
>
> **Volume tests**
> Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
>
> **Functional tests**
> Accuracy tests preformed on small size data.
>
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/
-----------------------------------------------------------
(Updated Sept. 5, 2019, 1:14 a.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include:
- Configuration enabled backing directory.
- Removed use of _AtlasRequest_ choosing backing store use.
Bugs: ATLAS-3396
https://issues.apache.org/jira/browse/ATLAS-3396
Repository: atlas
Description (updated)
-------
**Background**
The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
**Approach**
This is a backward compatible.
- - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
- Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
- Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
**CURL**
Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties.
_CURL_
```
curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
```
**Documentation**
Updated.
Diffs (updated)
-----
docs/src/site/twiki/Import-API-Options.twiki 4004e7013
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 6a9985642
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
Diff: https://reviews.apache.org/r/71426/diff/4/
Changes: https://reviews.apache.org/r/71426/diff/3-4/
Testing
-------
**Unit tests**
Updated to exercise the new implementation.
**Volume tests**
Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
**Functional tests**
Accuracy tests preformed on small size data.
**Pre-commit Build**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
Thanks,
Ashutosh Mestry
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/
-----------------------------------------------------------
(Updated Sept. 4, 2019, 7:58 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Upates include:
- Addressed review comments.
- _backingDirectory_ is now specified at application level.
Bugs: ATLAS-3396
https://issues.apache.org/jira/browse/ATLAS-3396
Repository: atlas
Description
-------
**Background**
The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
**Approach**
This is a backward compatible.
- - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
- Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
- Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
**CURL**
Use pre-configured temporary directory:
```
{
"options": {
"backingDirectory": "/grid/0/temp"
}
}
```
To use default temp directory:
```
{
"options": {
"backingDirectory": ""
}
}
``
_CURL_
```
curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
```
**Documentation**
Updated.
Diffs (updated)
-----
docs/src/site/twiki/Import-API-Options.twiki 4004e7013
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 6a9985642
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
Diff: https://reviews.apache.org/r/71426/diff/3/
Changes: https://reviews.apache.org/r/71426/diff/2-3/
Testing
-------
**Unit tests**
Updated to exercise the new implementation.
**Volume tests**
Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
**Functional tests**
Accuracy tests preformed on small size data.
**Pre-commit Build**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1379/console
Thanks,
Ashutosh Mestry
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
> On Sept. 3, 2019, 5:38 p.m., Madhan Neethiraj wrote:
> > docs/src/site/twiki/Import-API-Options.twiki
> > Lines 149 (patched)
> > <https://reviews.apache.org/r/71426/diff/2/?file=2163279#file2163279line149>
> >
> > - please consider avoiding the client to specify a directory to write at the server side. The location must be configured at the server-side.
> > - if use of 'backingDirectory' benefits all imports, can this approach be used for all imports, instead of having the client to request for it? In such case, consider creating a directory, perhaps named as a generated guid, under the configured location - to avoid potential reuse contents across multiple requests. Also, care must be taken to delete the contents after the import is completed - successfully or unsuccessfully.
My original implementation had it at as application properties. Reason I maintained this on a per-request basis is for backward compatibility and for us to be able to use in-memory for smaller imports.
Right now, backing directory is suffixed with a timestamp (_System.currentMilliseconds_) that way it is specific to each import.
About delete/clean-up: This is being done.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/#review217553
-----------------------------------------------------------
On Sept. 3, 2019, 5:24 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71426/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2019, 5:24 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3396
> https://issues.apache.org/jira/browse/ATLAS-3396
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
>
> **Approach**
> This is a backward compatible.
>
> - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
> - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
> - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
>
>
> **CURL**
>
> Use pre-configured temporary directory:
> ```
> {
> "options": {
> "backingDirectory": "/grid/0/temp"
> }
> }
> ```
>
> To use default temp directory:
>
> ```
> {
> "options": {
> "backingDirectory": ""
> }
> }
>
> ``
>
> _CURL_
> ```
> curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
> ```
>
> **Documentation**
> Updated.
>
>
> Diffs
> -----
>
> docs/src/site/twiki/Import-API-Options.twiki 4004e7013
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 6a9985642
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
>
>
> Diff: https://reviews.apache.org/r/71426/diff/2/
>
>
> Testing
> -------
>
> **Unit tests**
> Updated to exercise the new implementation.
>
> **Volume tests**
> Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
>
> **Functional tests**
> Accuracy tests preformed on small size data.
>
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1379/console
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/#review217553
-----------------------------------------------------------
docs/src/site/twiki/Import-API-Options.twiki
Lines 149 (patched)
<https://reviews.apache.org/r/71426/#comment304815>
- please consider avoiding the client to specify a directory to write at the server side. The location must be configured at the server-side.
- if use of 'backingDirectory' benefits all imports, can this approach be used for all imports, instead of having the client to request for it? In such case, consider creating a directory, perhaps named as a generated guid, under the configured location - to avoid potential reuse contents across multiple requests. Also, care must be taken to delete the contents after the import is completed - successfully or unsuccessfully.
- Madhan Neethiraj
On Sept. 3, 2019, 5:24 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71426/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2019, 5:24 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3396
> https://issues.apache.org/jira/browse/ATLAS-3396
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
>
> **Approach**
> This is a backward compatible.
>
> - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
> - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
> - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
>
>
> **CURL**
>
> Use pre-configured temporary directory:
> ```
> {
> "options": {
> "backingDirectory": "/grid/0/temp"
> }
> }
> ```
>
> To use default temp directory:
>
> ```
> {
> "options": {
> "backingDirectory": ""
> }
> }
>
> ``
>
> _CURL_
> ```
> curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
> ```
>
> **Documentation**
> Updated.
>
>
> Diffs
> -----
>
> docs/src/site/twiki/Import-API-Options.twiki 4004e7013
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 6a9985642
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
>
>
> Diff: https://reviews.apache.org/r/71426/diff/2/
>
>
> Testing
> -------
>
> **Unit tests**
> Updated to exercise the new implementation.
>
> **Volume tests**
> Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
>
> **Functional tests**
> Accuracy tests preformed on small size data.
>
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1379/console
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 71426: Import Service: Support for Backing
Directory for Large Imports
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/
-----------------------------------------------------------
(Updated Sept. 3, 2019, 5:24 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include: Minor refactoring.
Bugs: ATLAS-3396
https://issues.apache.org/jira/browse/ATLAS-3396
Repository: atlas
Description
-------
**Background**
The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported.
**Approach**
This is a backward compatible.
- - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file.
- Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_.
- Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import.
**CURL**
Use pre-configured temporary directory:
```
{
"options": {
"backingDirectory": "/grid/0/temp"
}
}
```
To use default temp directory:
```
{
"options": {
"backingDirectory": ""
}
}
``
_CURL_
```
curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
```
**Documentation**
Updated.
Diffs (updated)
-----
docs/src/site/twiki/Import-API-Options.twiki 4004e7013
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 6a9985642
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8
repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0
Diff: https://reviews.apache.org/r/71426/diff/2/
Changes: https://reviews.apache.org/r/71426/diff/1-2/
Testing
-------
**Unit tests**
Updated to exercise the new implementation.
**Volume tests**
Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation.
**Functional tests**
Accuracy tests preformed on small size data.
**Pre-commit Build**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1379/console
Thanks,
Ashutosh Mestry