You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Pinal Shah <pi...@freestoneinfotech.com> on 2020/09/03 10:26:15 UTC
Re: Review Request 72815: Deletion of non existing hive entities
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/
-----------------------------------------------------------
(Updated Sept. 3, 2020, 10:26 a.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Addressed review comments
Repository: atlas
Description
-------
**Problem:** Whenever database or table is dropped in hive, and HiveHook is not enabled, we dont have anyway to get the database and table sync with hive.
**Workaround:** Added support to delete hive entities in Atlas which are dropped in hive.
**Usage:** ./import-hive.sh -deleteNonExisting
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java e659ca041
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358
Diff: https://reviews.apache.org/r/72815/diff/2/
Changes: https://reviews.apache.org/r/72815/diff/1-2/
Testing
-------
Manually tested
Thanks,
Pinal Shah
Re: Review Request 72815: Deletion of non existing hive entities
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
> On Sept. 4, 2020, 4:04 p.m., Madhan Neethiraj wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
> > Lines 959 (patched)
> > <https://reviews.apache.org/r/72815/diff/2/?file=2239187#file2239187line968>
> >
> > Returning after deleting one entity? Please review.
thanks for pointing, it is incorrect
- Pinal
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/#review221806
-----------------------------------------------------------
On Sept. 5, 2020, 5:02 a.m., Pinal Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72815/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2020, 5:02 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Problem:** Whenever database or table is dropped in hive, and HiveHook is not enabled, we dont have anyway to get the database and table sync with hive.
>
> **Workaround:** Added support to delete hive entities in Atlas which are dropped in hive.
>
> **Usage:** ./import-hive.sh -deleteNonExisting
>
> **NOTE:** **atlas.hook.hive.page.limit** property is added to configure the pageSize/limit while fetching entities from Atlas
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java e659ca041
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358
>
>
> Diff: https://reviews.apache.org/r/72815/diff/3/
>
>
> Testing
> -------
>
> Manually tested
>
>
> Thanks,
>
> Pinal Shah
>
>
Re: Review Request 72815: Deletion of non existing hive entities
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/#review221806
-----------------------------------------------------------
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 862 (patched)
<https://reviews.apache.org/r/72815/#comment310888>
Since there is only one criteria in the filter, there is no need for innerFc. Consider the following:
SearchParameters.FilterCriteria fc = new SearchParameters.FilterCriteria();
fc.setAttributeName(ATTRIBUTE_CLUSTER_NAME);
fc.setAttributeValue(clusterName);
fc.setOperator(SearchParameters.Operator.EQ);
Also, the filter criteira is the same across all iteration of the for loop at #858. Consider setting up the criteira before once before entering the for loop.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 959 (patched)
<https://reviews.apache.org/r/72815/#comment310890>
Returning after deleting one entity? Please review.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 968 (patched)
<https://reviews.apache.org/r/72815/#comment310892>
deleteHiveMetadata() => deleteEntitiesForNonExistingHiveMetadata()
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 999 (patched)
<https://reviews.apache.org/r/72815/#comment310891>
There is only one parameter to replace, as LOG.error() will print execption details for 'e'. Please review and remove ', {}' at the end of the message.
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 1006 (patched)
<https://reviews.apache.org/r/72815/#comment310889>
Consider replacing:
Queue => List
LinkedList => ArrayList
- Madhan Neethiraj
On Sept. 3, 2020, 12:21 p.m., Pinal Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72815/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2020, 12:21 p.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Problem:** Whenever database or table is dropped in hive, and HiveHook is not enabled, we dont have anyway to get the database and table sync with hive.
>
> **Workaround:** Added support to delete hive entities in Atlas which are dropped in hive.
>
> **Usage:** ./import-hive.sh -deleteNonExisting
>
> **NOTE:** **atlas.hook.hive.page.limit** property is added to configure the pageSize/limit while fetching entities from Atlas
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java e659ca041
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358
>
>
> Diff: https://reviews.apache.org/r/72815/diff/2/
>
>
> Testing
> -------
>
> Manually tested
>
>
> Thanks,
>
> Pinal Shah
>
>
Re: Review Request 72815: Deletion of non existing hive entities
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/
-----------------------------------------------------------
(Updated Sept. 11, 2020, 12:27 p.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
updating patch for branch-0.8
Repository: atlas
Description
-------
**Problem:** Whenever database or table is dropped in hive, and HiveHook is not enabled, we dont have anyway to get the database and table sync with hive.
**Workaround:** Added support to delete hive entities in Atlas which are dropped in hive.
**Usage:** ./import-hive.sh -deleteNonExisting
**NOTE:** **atlas.hook.hive.page.limit** property is added to configure the pageSize/limit while fetching entities from Atlas
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java f18d01b7c
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358
Diff: https://reviews.apache.org/r/72815/diff/4/
Changes: https://reviews.apache.org/r/72815/diff/3-4/
Testing
-------
Manually tested
Thanks,
Pinal Shah
Re: Review Request 72815: Deletion of non existing hive entities
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/#review221808
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On Sept. 5, 2020, 5:02 a.m., Pinal Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72815/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2020, 5:02 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Problem:** Whenever database or table is dropped in hive, and HiveHook is not enabled, we dont have anyway to get the database and table sync with hive.
>
> **Workaround:** Added support to delete hive entities in Atlas which are dropped in hive.
>
> **Usage:** ./import-hive.sh -deleteNonExisting
>
> **NOTE:** **atlas.hook.hive.page.limit** property is added to configure the pageSize/limit while fetching entities from Atlas
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java e659ca041
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358
>
>
> Diff: https://reviews.apache.org/r/72815/diff/3/
>
>
> Testing
> -------
>
> Manually tested
>
>
> Thanks,
>
> Pinal Shah
>
>
Re: Review Request 72815: Deletion of non existing hive entities
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/
-----------------------------------------------------------
(Updated Sept. 5, 2020, 5:02 a.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
addressed comments
Repository: atlas
Description
-------
**Problem:** Whenever database or table is dropped in hive, and HiveHook is not enabled, we dont have anyway to get the database and table sync with hive.
**Workaround:** Added support to delete hive entities in Atlas which are dropped in hive.
**Usage:** ./import-hive.sh -deleteNonExisting
**NOTE:** **atlas.hook.hive.page.limit** property is added to configure the pageSize/limit while fetching entities from Atlas
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java e659ca041
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358
Diff: https://reviews.apache.org/r/72815/diff/3/
Changes: https://reviews.apache.org/r/72815/diff/2-3/
Testing
-------
Manually tested
Thanks,
Pinal Shah
Re: Review Request 72815: Deletion of non existing hive entities
Posted by Pinal Shah <pi...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/
-----------------------------------------------------------
(Updated Sept. 3, 2020, 12:21 p.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Repository: atlas
Description (updated)
-------
**Problem:** Whenever database or table is dropped in hive, and HiveHook is not enabled, we dont have anyway to get the database and table sync with hive.
**Workaround:** Added support to delete hive entities in Atlas which are dropped in hive.
**Usage:** ./import-hive.sh -deleteNonExisting
**NOTE:** **atlas.hook.hive.page.limit** property is added to configure the pageSize/limit while fetching entities from Atlas
Diffs
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java e659ca041
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358
Diff: https://reviews.apache.org/r/72815/diff/2/
Testing
-------
Manually tested
Thanks,
Pinal Shah