Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.

[Ashutosh Mestry <as...@gmail.com>]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/#review224552
-----------------------------------------------------------


Ship it!




Ship It!

- Ashutosh Mestry


On June 23, 2022, 6:52 a.m., Paresh Devalia wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74036/
> -----------------------------------------------------------
> 
> (Updated June 23, 2022, 6:52 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
> 
> 
> Bugs: ATLAS-4627
>     https://issues.apache.org/jira/browse/ATLAS-4627
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> PROBLEM STATEMENT
> 
> Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
> Created tables with 10 record in DB_1 and DB_2 .
> In Atlas UI we checked it has no entity created, so we ran import-hive.sh
> bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
> bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
> 
> Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
> 
> bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
> As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
> 
> 
> Diffs
> -----
> 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c 
> 
> 
> Diff: https://reviews.apache.org/r/74036/diff/1/
> 
> 
> Testing
> -------
> 
> Manual testing is done.
> 
> 1) mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
>   Atlas server runnning
> 2) Quick started was passed successfully
> 
> Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1130/consoleFull
> 
> Hive DB Testing 
> 
> 1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
> 2) Created tables with 10 record in DB_1 and DB_2 .
> 3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
> 4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
> 5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
> 6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas 
> 7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
> 8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
> 9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
> 10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
> 11) Checked the deleted table on statistics page in Atlas UI.
> 
> 
> Thanks,
> 
> Paresh Devalia
> 
>