You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Sidharth Mishra <si...@gmail.com> on 2021/09/15 23:00:43 UTC
Review Request 73586: ATLAS-4424: Enhance the Import hive utility to
create export zip files and run bulk import
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/
-----------------------------------------------------------
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4424
https://issues.apache.org/jira/browse/ATLAS-4424
Repository: atlas
Description
-------
ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
Diffs
-----
addons/hive-bridge/src/bin/import-hive.sh fde486f28
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 08471b480
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java d56261f78
repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java baa040f4c
Diff: https://reviews.apache.org/r/73586/diff/1/
Testing
-------
Manually tested the new zip file creation and bulk import to atlas
Thanks,
Sidharth Mishra
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/
-----------------------------------------------------------
(Updated Nov. 3, 2021, 11:45 p.m.)
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4424
https://issues.apache.org/jira/browse/ATLAS-4424
Repository: atlas
Description
-------
ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
Diffs (updated)
-----
addons/hive-bridge/src/bin/import-hive.sh c3539377f
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java bfb114814
repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
Diff: https://reviews.apache.org/r/73586/diff/8/
Changes: https://reviews.apache.org/r/73586/diff/7-8/
Testing
-------
Manually tested the new zip file creation and bulk import to atlas. PreCommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/941/
Thanks,
Sidharth Mishra
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/#review223712
-----------------------------------------------------------
Fix it, then Ship it!
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 459 (patched)
<https://reviews.apache.org/r/73586/#comment312820>
this method - loadRelatedGuidsFromAtlas() populates the 'qualifiedNameGuidMap' cache. Consider renaming to: populateQualifiedNameGuidMap() for more relavence.
- Sarath Subramanian
On Oct. 29, 2021, 12:40 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73586/
> -----------------------------------------------------------
>
> (Updated Oct. 29, 2021, 12:40 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4424
> https://issues.apache.org/jira/browse/ATLAS-4424
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/bin/import-hive.sh c3539377f
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
> intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java bfb114814
> repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
>
>
> Diff: https://reviews.apache.org/r/73586/diff/7/
>
>
> Testing
> -------
>
> Manually tested the new zip file creation and bulk import to atlas. PreCommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/941/
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/
-----------------------------------------------------------
(Updated Oct. 29, 2021, 7:40 p.m.)
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4424
https://issues.apache.org/jira/browse/ATLAS-4424
Repository: atlas
Description
-------
ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
Diffs
-----
addons/hive-bridge/src/bin/import-hive.sh c3539377f
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java bfb114814
repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
Diff: https://reviews.apache.org/r/73586/diff/7/
Testing (updated)
-------
Manually tested the new zip file creation and bulk import to atlas. PreCommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/941/
Thanks,
Sidharth Mishra
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/
-----------------------------------------------------------
(Updated Oct. 20, 2021, 4:49 p.m.)
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4424
https://issues.apache.org/jira/browse/ATLAS-4424
Repository: atlas
Description
-------
ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
Diffs (updated)
-----
addons/hive-bridge/src/bin/import-hive.sh c3539377f
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java bfb114814
repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
Diff: https://reviews.apache.org/r/73586/diff/7/
Changes: https://reviews.apache.org/r/73586/diff/6-7/
Testing
-------
Manually tested the new zip file creation and bulk import to atlas
Thanks,
Sidharth Mishra
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sidharth Mishra <si...@gmail.com>.
> On Oct. 18, 2021, 10:44 p.m., Sarath Subramanian wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
> > Line 131 (original), 150 (patched)
> > <https://reviews.apache.org/r/73586/diff/4/?file=2254089#file2254089line153>
> >
> > instead of throwing ParseException, should we set exitCode = EXIT_CODE_INVALID_ARG?
> >
> > Also add a lOG.ERROR about the details.
As we handle at catch block and print the LOG.error msg and set exit code accordingly.
> On Oct. 18, 2021, 10:44 p.m., Sarath Subramanian wrote:
> > addons/hive-bridge/src/bin/import-hive.sh
> > Line 160 (original), 168 (patched)
> > <https://reviews.apache.org/r/73586/diff/6/?file=2254123#file2254123line168>
> >
> > nit: this overrides previous change. Consider reverting and keep the original echo msg.
This to display the error msg only when the import is either successful or failed. Other than that like invalid arg prints different appropriate msg.
> On Oct. 18, 2021, 10:44 p.m., Sarath Subramanian wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
> > Lines 163 (patched)
> > <https://reviews.apache.org/r/73586/diff/6/?file=2254125#file2254125line163>
> >
> > line 163-169 (4 methods) deals with error checking and initializing. Consider encapsulating to an init() method to hide the specifics.
Put the initialization of zip stream into one as we need the try finally block for this (keeping the validateOutputFilePath separate)
- Sidharth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/#review223618
-----------------------------------------------------------
On Oct. 18, 2021, 6:18 a.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73586/
> -----------------------------------------------------------
>
> (Updated Oct. 18, 2021, 6:18 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4424
> https://issues.apache.org/jira/browse/ATLAS-4424
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/bin/import-hive.sh c3539377f
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
> intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java bfb114814
> repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
>
>
> Diff: https://reviews.apache.org/r/73586/diff/6/
>
>
> Testing
> -------
>
> Manually tested the new zip file creation and bulk import to atlas
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/#review223618
-----------------------------------------------------------
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Line 131 (original), 150 (patched)
<https://reviews.apache.org/r/73586/#comment312721>
instead of throwing ParseException, should we set exitCode = EXIT_CODE_INVALID_ARG?
Also add a lOG.ERROR about the details.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 190 (patched)
<https://reviews.apache.org/r/73586/#comment312722>
consider combining catch statements 190 and 194, since the contents are same:
catch (MissingArgumentException | ParseException e) {
...
}
addons/hive-bridge/src/bin/import-hive.sh
Line 160 (original), 168 (patched)
<https://reviews.apache.org/r/73586/#comment312745>
nit: this overrides previous change. Consider reverting and keep the original echo msg.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 163 (patched)
<https://reviews.apache.org/r/73586/#comment312744>
line 163-169 (4 methods) deals with error checking and initializing. Consider encapsulating to an init() method to hide the specifics.
intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java
Line 38 (original), 33 (patched)
<https://reviews.apache.org/r/73586/#comment312746>
instead of getting rid of "name" and using "fileHash". Can we capture both - name and fileHash?
will this change in POJO, affect cross-version compat (backward-compatible)?
- Sarath Subramanian
On Oct. 17, 2021, 11:18 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73586/
> -----------------------------------------------------------
>
> (Updated Oct. 17, 2021, 11:18 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4424
> https://issues.apache.org/jira/browse/ATLAS-4424
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/bin/import-hive.sh c3539377f
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
> intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java bfb114814
> repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
>
>
> Diff: https://reviews.apache.org/r/73586/diff/6/
>
>
> Testing
> -------
>
> Manually tested the new zip file creation and bulk import to atlas
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/
-----------------------------------------------------------
(Updated Oct. 18, 2021, 6:18 a.m.)
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4424
https://issues.apache.org/jira/browse/ATLAS-4424
Repository: atlas
Description
-------
ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
Diffs (updated)
-----
addons/hive-bridge/src/bin/import-hive.sh c3539377f
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java bfb114814
repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
Diff: https://reviews.apache.org/r/73586/diff/6/
Changes: https://reviews.apache.org/r/73586/diff/5-6/
Testing
-------
Manually tested the new zip file creation and bulk import to atlas
Thanks,
Sidharth Mishra
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/
-----------------------------------------------------------
(Updated Oct. 18, 2021, 6:03 a.m.)
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4424
https://issues.apache.org/jira/browse/ATLAS-4424
Repository: atlas
Description
-------
ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
Diffs (updated)
-----
addons/hive-bridge/src/bin/import-hive.sh c3539377f
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java bfb114814
repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
Diff: https://reviews.apache.org/r/73586/diff/5/
Changes: https://reviews.apache.org/r/73586/diff/4-5/
Testing
-------
Manually tested the new zip file creation and bulk import to atlas
Thanks,
Sidharth Mishra
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/
-----------------------------------------------------------
(Updated Oct. 13, 2021, 7:56 p.m.)
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4424
https://issues.apache.org/jira/browse/ATLAS-4424
Repository: atlas
Description
-------
ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
Diffs (updated)
-----
addons/hive-bridge/src/bin/import-hive.sh c3539377f
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java bfb114814
repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
Diff: https://reviews.apache.org/r/73586/diff/4/
Changes: https://reviews.apache.org/r/73586/diff/3-4/
Testing
-------
Manually tested the new zip file creation and bulk import to atlas
Thanks,
Sidharth Mishra
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/
-----------------------------------------------------------
(Updated Oct. 13, 2021, 5:50 p.m.)
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4424
https://issues.apache.org/jira/browse/ATLAS-4424
Repository: atlas
Description
-------
ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
Diffs (updated)
-----
addons/hive-bridge/src/bin/import-hive.sh c3539377f
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java bfb114814
repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
Diff: https://reviews.apache.org/r/73586/diff/3/
Changes: https://reviews.apache.org/r/73586/diff/2-3/
Testing
-------
Manually tested the new zip file creation and bulk import to atlas
Thanks,
Sidharth Mishra
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/
-----------------------------------------------------------
(Updated Oct. 7, 2021, 11:13 p.m.)
Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
Bugs: ATLAS-4424
https://issues.apache.org/jira/browse/ATLAS-4424
Repository: atlas
Description
-------
ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
Diffs (updated)
-----
addons/hive-bridge/src/bin/import-hive.sh c3539377f
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
pom.xml 84630e76b
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java d56261f78
repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
Diff: https://reviews.apache.org/r/73586/diff/2/
Changes: https://reviews.apache.org/r/73586/diff/1-2/
Testing
-------
Manually tested the new zip file creation and bulk import to atlas
Thanks,
Sidharth Mishra
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sidharth Mishra <si...@gmail.com>.
> On Oct. 1, 2021, 10:58 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
> > Lines 90 (patched)
> > <https://reviews.apache.org/r/73586/diff/1/?file=2252834#file2252834line90>
> >
> > What is the reason for this?
For keeping the file hash as key instead of file name
> On Oct. 1, 2021, 10:58 p.m., Ashutosh Mestry wrote:
> > repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java
> > Line 46 (original), 48 (patched)
> > <https://reviews.apache.org/r/73586/diff/1/?file=2252835#file2252835line49>
> >
> > What is the reason for doing this?
This is for file hash as the key
- Sidharth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/#review223551
-----------------------------------------------------------
On Oct. 20, 2021, 4:49 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73586/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2021, 4:49 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4424
> https://issues.apache.org/jira/browse/ATLAS-4424
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/bin/import-hive.sh c3539377f
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
> intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java bfb114814
> repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
>
>
> Diff: https://reviews.apache.org/r/73586/diff/7/
>
>
> Testing
> -------
>
> Manually tested the new zip file creation and bulk import to atlas. PreCommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/941/
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/#review223551
-----------------------------------------------------------
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 130 (patched)
<https://reviews.apache.org/r/73586/#comment312629>
Rename: qualifiedNameGuidMap
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 300 (patched)
<https://reviews.apache.org/r/73586/#comment312626>
A get in to map with db as key should simplify this.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 300 (patched)
<https://reviews.apache.org/r/73586/#comment312627>
A get in to map with db as key should simplify this.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 415 (patched)
<https://reviews.apache.org/r/73586/#comment312628>
This will make large number of calls to Atlas thereby slowing the operation down.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 529 (patched)
<https://reviews.apache.org/r/73586/#comment312630>
This method converts Table to AtlasEntity and writes to stream. Consider separating it in 2 distinct methods. Say
AtlasEntityWithExtInfo toAtlasEntity(Table table) and write(OutputStream stream, AtlasEntityWithExtInfo entity)
The write method will be reused by all other operations.
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Line 56 (original)
<https://reviews.apache.org/r/73586/#comment312620>
This is not necessary.
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Lines 90 (patched)
<https://reviews.apache.org/r/73586/#comment312623>
What is the reason for this?
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Lines 180 (patched)
<https://reviews.apache.org/r/73586/#comment312622>
This is not necessary.
repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java
Line 46 (original), 48 (patched)
<https://reviews.apache.org/r/73586/#comment312621>
What is the reason for doing this?
- Ashutosh Mestry
On Sept. 15, 2021, 11 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73586/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2021, 11 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4424
> https://issues.apache.org/jira/browse/ATLAS-4424
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/bin/import-hive.sh fde486f28
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 08471b480
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
> intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java d56261f78
> repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java baa040f4c
>
>
> Diff: https://reviews.apache.org/r/73586/diff/1/
>
>
> Testing
> -------
>
> Manually tested the new zip file creation and bulk import to atlas
>
>
> Thanks,
>
> Sidharth Mishra
>
>