You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Paresh Devalia <pa...@freestoneinfotech.com> on 2022/08/04 06:53:54 UTC
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/
-----------------------------------------------------------
(Updated Aug. 4, 2022, 6:53 a.m.)
Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
Bugs: ATLAS-4627
https://issues.apache.org/jira/browse/ATLAS-4627
Repository: atlas
Description
-------
PROBLEM STATEMENT
Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
Created tables with 10 record in DB_1 and DB_2 .
In Atlas UI we checked it has no entity created, so we ran import-hive.sh
bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java ae7ab1a22
Diff: https://reviews.apache.org/r/74036/diff/2/
Changes: https://reviews.apache.org/r/74036/diff/1-2/
Testing (updated)
-------
Manual testing is done, also testcase running successfully.
1) mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
Atlas server runnning
2) Quick started was passed successfully
Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1181/console
Hive DB Testing
1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
2) Created tables with 10 record in DB_1 and DB_2 .
3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas
7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
11) Checked the deleted table on statistics page in Atlas UI.
Thanks,
Paresh Devalia
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
Posted by Paresh Devalia <pa...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/
-----------------------------------------------------------
(Updated Sept. 8, 2022, 9:02 a.m.)
Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
Bugs: ATLAS-4627
https://issues.apache.org/jira/browse/ATLAS-4627
Repository: atlas
Description
-------
PROBLEM STATEMENT
Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
Created tables with 10 record in DB_1 and DB_2 .
In Atlas UI we checked it has no entity created, so we ran import-hive.sh
bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
Diffs
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java ae7ab1a22
Diff: https://reviews.apache.org/r/74036/diff/7/
Testing (updated)
-------
Manual testing is done, also testcase running successfully.
1) mvn clean package -Pdist,embedded-hbase-solr
Atlas server runnning
2) Quick started was passed successfully
Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1209/console
Hive DB Testing
1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
2) Created tables with 10 record in DB_1 and DB_2 .
3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas
7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
11) Checked the deleted table on statistics page in Atlas UI.
Thanks,
Paresh Devalia
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/#review224650
-----------------------------------------------------------
Ship it!
Ship It!
- Pinal Shah
On Sept. 7, 2022, 1:02 p.m., Paresh Devalia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74036/
> -----------------------------------------------------------
>
> (Updated Sept. 7, 2022, 1:02 p.m.)
>
>
> Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
>
>
> Bugs: ATLAS-4627
> https://issues.apache.org/jira/browse/ATLAS-4627
>
>
> Repository: atlas
>
>
> Description
> -------
>
> PROBLEM STATEMENT
>
> Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
> Created tables with 10 record in DB_1 and DB_2 .
> In Atlas UI we checked it has no entity created, so we ran import-hive.sh
> bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
> bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
>
> Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
>
> bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
> As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java ae7ab1a22
>
>
> Diff: https://reviews.apache.org/r/74036/diff/7/
>
>
> Testing
> -------
>
> Manual testing is done, also testcase running successfully.
>
> 1) mvn clean package -Pdist,embedded-hbase-solr
> Atlas server runnning
> 2) Quick started was passed successfully
>
> Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1204/console
>
> Hive DB Testing
>
> 1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
> 2) Created tables with 10 record in DB_1 and DB_2 .
> 3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
> 4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
> 5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
> 6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas
> 7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
> 8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
> 9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
> 10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
> 11) Checked the deleted table on statistics page in Atlas UI.
>
>
> Thanks,
>
> Paresh Devalia
>
>
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
Posted by Paresh Devalia <pa...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/
-----------------------------------------------------------
(Updated Sept. 7, 2022, 1:02 p.m.)
Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
Bugs: ATLAS-4627
https://issues.apache.org/jira/browse/ATLAS-4627
Repository: atlas
Description
-------
PROBLEM STATEMENT
Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
Created tables with 10 record in DB_1 and DB_2 .
In Atlas UI we checked it has no entity created, so we ran import-hive.sh
bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java ae7ab1a22
Diff: https://reviews.apache.org/r/74036/diff/7/
Changes: https://reviews.apache.org/r/74036/diff/6-7/
Testing
-------
Manual testing is done, also testcase running successfully.
1) mvn clean package -Pdist,embedded-hbase-solr
Atlas server runnning
2) Quick started was passed successfully
Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1204/console
Hive DB Testing
1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
2) Created tables with 10 record in DB_1 and DB_2 .
3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas
7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
11) Checked the deleted table on statistics page in Atlas UI.
Thanks,
Paresh Devalia
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/#review224649
-----------------------------------------------------------
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Line 1193 (original), 1204 (patched)
<https://reviews.apache.org/r/74036/#comment313455>
Can you put this inside else condition?
- Pinal Shah
On Sept. 7, 2022, 12:45 p.m., Paresh Devalia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74036/
> -----------------------------------------------------------
>
> (Updated Sept. 7, 2022, 12:45 p.m.)
>
>
> Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
>
>
> Bugs: ATLAS-4627
> https://issues.apache.org/jira/browse/ATLAS-4627
>
>
> Repository: atlas
>
>
> Description
> -------
>
> PROBLEM STATEMENT
>
> Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
> Created tables with 10 record in DB_1 and DB_2 .
> In Atlas UI we checked it has no entity created, so we ran import-hive.sh
> bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
> bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
>
> Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
>
> bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
> As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java ae7ab1a22
>
>
> Diff: https://reviews.apache.org/r/74036/diff/6/
>
>
> Testing
> -------
>
> Manual testing is done, also testcase running successfully.
>
> 1) mvn clean package -Pdist,embedded-hbase-solr
> Atlas server runnning
> 2) Quick started was passed successfully
>
> Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1204/console
>
> Hive DB Testing
>
> 1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
> 2) Created tables with 10 record in DB_1 and DB_2 .
> 3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
> 4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
> 5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
> 6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas
> 7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
> 8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
> 9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
> 10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
> 11) Checked the deleted table on statistics page in Atlas UI.
>
>
> Thanks,
>
> Paresh Devalia
>
>
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
Posted by Paresh Devalia <pa...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/
-----------------------------------------------------------
(Updated Sept. 7, 2022, 12:45 p.m.)
Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
Bugs: ATLAS-4627
https://issues.apache.org/jira/browse/ATLAS-4627
Repository: atlas
Description
-------
PROBLEM STATEMENT
Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
Created tables with 10 record in DB_1 and DB_2 .
In Atlas UI we checked it has no entity created, so we ran import-hive.sh
bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java ae7ab1a22
Diff: https://reviews.apache.org/r/74036/diff/6/
Changes: https://reviews.apache.org/r/74036/diff/5-6/
Testing
-------
Manual testing is done, also testcase running successfully.
1) mvn clean package -Pdist,embedded-hbase-solr
Atlas server runnning
2) Quick started was passed successfully
Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1204/console
Hive DB Testing
1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
2) Created tables with 10 record in DB_1 and DB_2 .
3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas
7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
11) Checked the deleted table on statistics page in Atlas UI.
Thanks,
Paresh Devalia
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
Posted by Paresh Devalia <pa...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/
-----------------------------------------------------------
(Updated Sept. 7, 2022, 11:27 a.m.)
Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
Bugs: ATLAS-4627
https://issues.apache.org/jira/browse/ATLAS-4627
Repository: atlas
Description
-------
PROBLEM STATEMENT
Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
Created tables with 10 record in DB_1 and DB_2 .
In Atlas UI we checked it has no entity created, so we ran import-hive.sh
bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java ae7ab1a22
Diff: https://reviews.apache.org/r/74036/diff/5/
Changes: https://reviews.apache.org/r/74036/diff/4-5/
Testing
-------
Manual testing is done, also testcase running successfully.
1) mvn clean package -Pdist,embedded-hbase-solr
Atlas server runnning
2) Quick started was passed successfully
Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1204/console
Hive DB Testing
1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
2) Created tables with 10 record in DB_1 and DB_2 .
3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas
7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
11) Checked the deleted table on statistics page in Atlas UI.
Thanks,
Paresh Devalia
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/#review224646
-----------------------------------------------------------
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Line 1170 (original), 1176 (patched)
<https://reviews.apache.org/r/74036/#comment313451>
Put this log inside else condition
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 1281 (patched)
<https://reviews.apache.org/r/74036/#comment313452>
Can we log, db qualifiedName here?
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 1284 (patched)
<https://reviews.apache.org/r/74036/#comment313449>
is this count needed?
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 1285 (patched)
<https://reviews.apache.org/r/74036/#comment313450>
As you are finding a particular database, will there be more than one db?
Is this log needed?
- Pinal Shah
On Sept. 6, 2022, 7:21 a.m., Paresh Devalia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74036/
> -----------------------------------------------------------
>
> (Updated Sept. 6, 2022, 7:21 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
>
>
> Bugs: ATLAS-4627
> https://issues.apache.org/jira/browse/ATLAS-4627
>
>
> Repository: atlas
>
>
> Description
> -------
>
> PROBLEM STATEMENT
>
> Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
> Created tables with 10 record in DB_1 and DB_2 .
> In Atlas UI we checked it has no entity created, so we ran import-hive.sh
> bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
> bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
>
> Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
>
> bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
> As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java ae7ab1a22
>
>
> Diff: https://reviews.apache.org/r/74036/diff/4/
>
>
> Testing
> -------
>
> Manual testing is done, also testcase running successfully.
>
> 1) mvn clean package -Pdist,embedded-hbase-solr
> Atlas server runnning
> 2) Quick started was passed successfully
>
> Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1204/console
>
> Hive DB Testing
>
> 1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
> 2) Created tables with 10 record in DB_1 and DB_2 .
> 3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
> 4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
> 5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
> 6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas
> 7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
> 8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
> 9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
> 10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
> 11) Checked the deleted table on statistics page in Atlas UI.
>
>
> Thanks,
>
> Paresh Devalia
>
>
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/#review224647
-----------------------------------------------------------
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 1298 (patched)
<https://reviews.apache.org/r/74036/#comment313453>
Need to address all the above comments in this method
- Pinal Shah
On Sept. 6, 2022, 7:21 a.m., Paresh Devalia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74036/
> -----------------------------------------------------------
>
> (Updated Sept. 6, 2022, 7:21 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
>
>
> Bugs: ATLAS-4627
> https://issues.apache.org/jira/browse/ATLAS-4627
>
>
> Repository: atlas
>
>
> Description
> -------
>
> PROBLEM STATEMENT
>
> Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
> Created tables with 10 record in DB_1 and DB_2 .
> In Atlas UI we checked it has no entity created, so we ran import-hive.sh
> bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
> bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
>
> Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
>
> bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
> As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java ae7ab1a22
>
>
> Diff: https://reviews.apache.org/r/74036/diff/4/
>
>
> Testing
> -------
>
> Manual testing is done, also testcase running successfully.
>
> 1) mvn clean package -Pdist,embedded-hbase-solr
> Atlas server runnning
> 2) Quick started was passed successfully
>
> Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1204/console
>
> Hive DB Testing
>
> 1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
> 2) Created tables with 10 record in DB_1 and DB_2 .
> 3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
> 4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
> 5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
> 6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas
> 7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
> 8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
> 9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
> 10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
> 11) Checked the deleted table on statistics page in Atlas UI.
>
>
> Thanks,
>
> Paresh Devalia
>
>
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
Posted by Paresh Devalia <pa...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/
-----------------------------------------------------------
(Updated Sept. 6, 2022, 7:21 a.m.)
Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
Bugs: ATLAS-4627
https://issues.apache.org/jira/browse/ATLAS-4627
Repository: atlas
Description
-------
PROBLEM STATEMENT
Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
Created tables with 10 record in DB_1 and DB_2 .
In Atlas UI we checked it has no entity created, so we ran import-hive.sh
bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java ae7ab1a22
Diff: https://reviews.apache.org/r/74036/diff/4/
Changes: https://reviews.apache.org/r/74036/diff/3-4/
Testing (updated)
-------
Manual testing is done, also testcase running successfully.
1) mvn clean package -Pdist,embedded-hbase-solr
Atlas server runnning
2) Quick started was passed successfully
Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1204/console
Hive DB Testing
1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
2) Created tables with 10 record in DB_1 and DB_2 .
3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas
7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
11) Checked the deleted table on statistics page in Atlas UI.
Thanks,
Paresh Devalia
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
Posted by Paresh Devalia <pa...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/
-----------------------------------------------------------
(Updated Aug. 4, 2022, 7:02 a.m.)
Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
Bugs: ATLAS-4627
https://issues.apache.org/jira/browse/ATLAS-4627
Repository: atlas
Description
-------
PROBLEM STATEMENT
Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
Created tables with 10 record in DB_1 and DB_2 .
In Atlas UI we checked it has no entity created, so we ran import-hive.sh
bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
Diffs
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java ae7ab1a22
Diff: https://reviews.apache.org/r/74036/diff/3/
Testing (updated)
-------
Manual testing is done, also testcase running successfully.
1) mvn clean package -Pdist,embedded-hbase-solr
Atlas server runnning
2) Quick started was passed successfully
Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1181/console
Hive DB Testing
1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
2) Created tables with 10 record in DB_1 and DB_2 .
3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas
7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
11) Checked the deleted table on statistics page in Atlas UI.
Thanks,
Paresh Devalia
Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.
Posted by Paresh Devalia <pa...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/
-----------------------------------------------------------
(Updated Aug. 4, 2022, 7 a.m.)
Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
Bugs: ATLAS-4627
https://issues.apache.org/jira/browse/ATLAS-4627
Repository: atlas
Description
-------
PROBLEM STATEMENT
Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
Created tables with 10 record in DB_1 and DB_2 .
In Atlas UI we checked it has no entity created, so we ran import-hive.sh
bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas .
bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete all the table from all DB which are present in Atlas and dropped from Hive.
As we have mention -d and -t argument in command for DB and table wise deleting, but still it ran for all DB and Tables.
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 28365bc5c
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java ae7ab1a22
Diff: https://reviews.apache.org/r/74036/diff/3/
Changes: https://reviews.apache.org/r/74036/diff/2-3/
Testing
-------
Manual testing is done, also testcase running successfully.
1) mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
Atlas server runnning
2) Quick started was passed successfully
Pre-commit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1181/console
Hive DB Testing
1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and hive_tez.
2) Created tables with 10 record in DB_1 and DB_2 .
3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create one entity in atlas.
5) bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as hooks are disable, so we will ran import-hive command to delete from atlas
7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will delete only table_1 from DB_1 which are present in Atlas and dropped from Hive.
8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command will delete only table_1 table from DB_2 which are present in Atlas and dropped from Hive.
9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete all table from DB_1 which are present in Atlas and dropped from Hive.
10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete all table from DB_2 which are present in Atlas and dropped from Hive.
11) Checked the deleted table on statistics page in Atlas UI.
Thanks,
Paresh Devalia