You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry via Review Board <no...@reviews.apache.org> on 2021/10/01 22:58:06 UTC
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/#review223551
-----------------------------------------------------------
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 130 (patched)
<https://reviews.apache.org/r/73586/#comment312629>
Rename: qualifiedNameGuidMap
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 300 (patched)
<https://reviews.apache.org/r/73586/#comment312626>
A get in to map with db as key should simplify this.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 300 (patched)
<https://reviews.apache.org/r/73586/#comment312627>
A get in to map with db as key should simplify this.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 415 (patched)
<https://reviews.apache.org/r/73586/#comment312628>
This will make large number of calls to Atlas thereby slowing the operation down.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 529 (patched)
<https://reviews.apache.org/r/73586/#comment312630>
This method converts Table to AtlasEntity and writes to stream. Consider separating it in 2 distinct methods. Say
AtlasEntityWithExtInfo toAtlasEntity(Table table) and write(OutputStream stream, AtlasEntityWithExtInfo entity)
The write method will be reused by all other operations.
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Line 56 (original)
<https://reviews.apache.org/r/73586/#comment312620>
This is not necessary.
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Lines 90 (patched)
<https://reviews.apache.org/r/73586/#comment312623>
What is the reason for this?
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Lines 180 (patched)
<https://reviews.apache.org/r/73586/#comment312622>
This is not necessary.
repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java
Line 46 (original), 48 (patched)
<https://reviews.apache.org/r/73586/#comment312621>
What is the reason for doing this?
- Ashutosh Mestry
On Sept. 15, 2021, 11 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73586/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2021, 11 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4424
> https://issues.apache.org/jira/browse/ATLAS-4424
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/bin/import-hive.sh fde486f28
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 08471b480
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
> intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java d56261f78
> repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java baa040f4c
>
>
> Diff: https://reviews.apache.org/r/73586/diff/1/
>
>
> Testing
> -------
>
> Manually tested the new zip file creation and bulk import to atlas
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility
to create export zip files and run bulk import
Posted by Sidharth Mishra <si...@gmail.com>.
> On Oct. 1, 2021, 10:58 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
> > Lines 90 (patched)
> > <https://reviews.apache.org/r/73586/diff/1/?file=2252834#file2252834line90>
> >
> > What is the reason for this?
For keeping the file hash as key instead of file name
> On Oct. 1, 2021, 10:58 p.m., Ashutosh Mestry wrote:
> > repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java
> > Line 46 (original), 48 (patched)
> > <https://reviews.apache.org/r/73586/diff/1/?file=2252835#file2252835line49>
> >
> > What is the reason for doing this?
This is for file hash as the key
- Sidharth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/#review223551
-----------------------------------------------------------
On Oct. 20, 2021, 4:49 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73586/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2021, 4:49 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4424
> https://issues.apache.org/jira/browse/ATLAS-4424
>
>
> Repository: atlas
>
>
> Description
> -------
>
> ATLAS-4424: Enhanced the Import hive utility to create export zip files and run bulk import
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/bin/import-hive.sh c3539377f
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java c361ac69a
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 2c1870444
> intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java 3430fdae8
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java a22c68790
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java bfb114814
> repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java f1dc990e8
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36
>
>
> Diff: https://reviews.apache.org/r/73586/diff/7/
>
>
> Testing
> -------
>
> Manually tested the new zip file creation and bulk import to atlas. PreCommit - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/941/
>
>
> Thanks,
>
> Sidharth Mishra
>
>