You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry via Review Board <no...@reviews.apache.org> on 2020/02/28 21:50:13 UTC
Review Request 72177: Import Service: Support zipDirect Format
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72177/
-----------------------------------------------------------
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Bugs: ATLAS-3641
https://issues.apache.org/jira/browse/ATLAS-3641
Repository: atlas
Description
-------
**Approach**
During genration entities are stores as 1 zip entry, _entities.json_.
New: _ZipSourceDirect_ contains a light-weight JSON parser to read the _entities.json_. This parser drives _next_ and _hasNext_ methods.
Modified: _AtlasImportRequest_ adds new key and value to specify import format.
Modified: _ImportService_ supports handling of this format.
Diffs
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 0b3ede93f
intg/src/main/java/org/apache/atlas/utils/AtlasJson.java 1c13860cf
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 27001e3a9
repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b47536
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43
repository/src/test/java/org/apache/atlas/repository/impexp/ZipDirectTest.java PRE-CREATION
repository/src/test/resources/zip-direct-1.zip PRE-CREATION
repository/src/test/resources/zip-direct-2.zip PRE-CREATION
repository/src/test/resources/zip-direct-3.zip PRE-CREATION
Diff: https://reviews.apache.org/r/72177/diff/1/
Testing
-------
**Unit tests**
- Tests for reading this format.
- _ImportServiceTest_ performs end-to-end test.
**Volume test**
Files with this format lends itself to volume. File with 10 million entities was performed.
Thanks,
Ashutosh Mestry
Re: Review Request 72177: Import Service: Support zipDirect Format
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72177/
-----------------------------------------------------------
(Updated March 2, 2020, 6:18 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include: New PC build#.
Bugs: ATLAS-3641
https://issues.apache.org/jira/browse/ATLAS-3641
Repository: atlas
Description
-------
**Approach**
During genration entities are stores as 1 zip entry, _entities.json_.
New: _ZipSourceDirect_ contains a light-weight JSON parser to read the _entities.json_. This parser drives _next_ and _hasNext_ methods.
Modified: _AtlasImportRequest_ adds new key and value to specify import format.
Modified: _ImportService_ supports handling of this format.
_import-options.json_
```
{
"options": {
"format": "zipDirect"
}
}
```
Diffs
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 0b3ede93f
intg/src/main/java/org/apache/atlas/utils/AtlasJson.java 1c13860cf
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 27001e3a9
repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b47536
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43
repository/src/test/java/org/apache/atlas/repository/impexp/ZipDirectTest.java PRE-CREATION
repository/src/test/resources/zip-direct-1.zip PRE-CREATION
repository/src/test/resources/zip-direct-2.zip PRE-CREATION
repository/src/test/resources/zip-direct-3.zip PRE-CREATION
Diff: https://reviews.apache.org/r/72177/diff/1/
Testing (updated)
-------
**Unit tests**
- Tests for reading this format.
- _ImportServiceTest_ performs end-to-end test.
**Volume test**
Files with this format lends itself to volume. File with 10 million entities was performed.
**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1693/
Thanks,
Ashutosh Mestry
Re: Review Request 72177: Import Service: Support zipDirect Format
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72177/#review219714
-----------------------------------------------------------
Ship it!
Ship It!
- Nikhil Bonte
On Feb. 28, 2020, 9:52 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72177/
> -----------------------------------------------------------
>
> (Updated Feb. 28, 2020, 9:52 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3641
> https://issues.apache.org/jira/browse/ATLAS-3641
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> During genration entities are stores as 1 zip entry, _entities.json_.
>
> New: _ZipSourceDirect_ contains a light-weight JSON parser to read the _entities.json_. This parser drives _next_ and _hasNext_ methods.
> Modified: _AtlasImportRequest_ adds new key and value to specify import format.
> Modified: _ImportService_ supports handling of this format.
>
> _import-options.json_
> ```
> {
> "options": {
> "format": "zipDirect"
> }
> }
> ```
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 0b3ede93f
> intg/src/main/java/org/apache/atlas/utils/AtlasJson.java 1c13860cf
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 27001e3a9
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b47536
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipDirectTest.java PRE-CREATION
> repository/src/test/resources/zip-direct-1.zip PRE-CREATION
> repository/src/test/resources/zip-direct-2.zip PRE-CREATION
> repository/src/test/resources/zip-direct-3.zip PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/72177/diff/1/
>
>
> Testing
> -------
>
> **Unit tests**
> - Tests for reading this format.
> - _ImportServiceTest_ performs end-to-end test.
>
> **Volume test**
> Files with this format lends itself to volume. File with 10 million entities was performed.
>
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 72177: Import Service: Support zipDirect Format
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72177/#review219715
-----------------------------------------------------------
"Ship It" for the patch.
Please review PC build - https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1686/console
Issue while applying patch as it has zips, I suggest to push zips as separate commit.
- Nikhil Bonte
On Feb. 28, 2020, 9:52 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72177/
> -----------------------------------------------------------
>
> (Updated Feb. 28, 2020, 9:52 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3641
> https://issues.apache.org/jira/browse/ATLAS-3641
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> During genration entities are stores as 1 zip entry, _entities.json_.
>
> New: _ZipSourceDirect_ contains a light-weight JSON parser to read the _entities.json_. This parser drives _next_ and _hasNext_ methods.
> Modified: _AtlasImportRequest_ adds new key and value to specify import format.
> Modified: _ImportService_ supports handling of this format.
>
> _import-options.json_
> ```
> {
> "options": {
> "format": "zipDirect"
> }
> }
> ```
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 0b3ede93f
> intg/src/main/java/org/apache/atlas/utils/AtlasJson.java 1c13860cf
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 27001e3a9
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b47536
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipDirectTest.java PRE-CREATION
> repository/src/test/resources/zip-direct-1.zip PRE-CREATION
> repository/src/test/resources/zip-direct-2.zip PRE-CREATION
> repository/src/test/resources/zip-direct-3.zip PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/72177/diff/1/
>
>
> Testing
> -------
>
> **Unit tests**
> - Tests for reading this format.
> - _ImportServiceTest_ performs end-to-end test.
>
> **Volume test**
> Files with this format lends itself to volume. File with 10 million entities was performed.
>
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 72177: Import Service: Support zipDirect Format
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72177/#review219717
-----------------------------------------------------------
Ship it!
Ship It!
- Nixon Rodrigues
On Feb. 28, 2020, 9:52 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72177/
> -----------------------------------------------------------
>
> (Updated Feb. 28, 2020, 9:52 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3641
> https://issues.apache.org/jira/browse/ATLAS-3641
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> During genration entities are stores as 1 zip entry, _entities.json_.
>
> New: _ZipSourceDirect_ contains a light-weight JSON parser to read the _entities.json_. This parser drives _next_ and _hasNext_ methods.
> Modified: _AtlasImportRequest_ adds new key and value to specify import format.
> Modified: _ImportService_ supports handling of this format.
>
> _import-options.json_
> ```
> {
> "options": {
> "format": "zipDirect"
> }
> }
> ```
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 0b3ede93f
> intg/src/main/java/org/apache/atlas/utils/AtlasJson.java 1c13860cf
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 27001e3a9
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b47536
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipDirectTest.java PRE-CREATION
> repository/src/test/resources/zip-direct-1.zip PRE-CREATION
> repository/src/test/resources/zip-direct-2.zip PRE-CREATION
> repository/src/test/resources/zip-direct-3.zip PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/72177/diff/1/
>
>
> Testing
> -------
>
> **Unit tests**
> - Tests for reading this format.
> - _ImportServiceTest_ performs end-to-end test.
>
> **Volume test**
> Files with this format lends itself to volume. File with 10 million entities was performed.
>
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 72177: Import Service: Support zipDirect Format
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72177/
-----------------------------------------------------------
(Updated Feb. 28, 2020, 9:52 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include: PC build and documentation.
Bugs: ATLAS-3641
https://issues.apache.org/jira/browse/ATLAS-3641
Repository: atlas
Description (updated)
-------
**Approach**
During genration entities are stores as 1 zip entry, _entities.json_.
New: _ZipSourceDirect_ contains a light-weight JSON parser to read the _entities.json_. This parser drives _next_ and _hasNext_ methods.
Modified: _AtlasImportRequest_ adds new key and value to specify import format.
Modified: _ImportService_ supports handling of this format.
_import-options.json_
```
{
"options": {
"format": "zipDirect"
}
}
```
Diffs
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 0b3ede93f
intg/src/main/java/org/apache/atlas/utils/AtlasJson.java 1c13860cf
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 27001e3a9
repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b47536
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43
repository/src/test/java/org/apache/atlas/repository/impexp/ZipDirectTest.java PRE-CREATION
repository/src/test/resources/zip-direct-1.zip PRE-CREATION
repository/src/test/resources/zip-direct-2.zip PRE-CREATION
repository/src/test/resources/zip-direct-3.zip PRE-CREATION
Diff: https://reviews.apache.org/r/72177/diff/1/
Testing (updated)
-------
**Unit tests**
- Tests for reading this format.
- _ImportServiceTest_ performs end-to-end test.
**Volume test**
Files with this format lends itself to volume. File with 10 million entities was performed.
**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/
Thanks,
Ashutosh Mestry