You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by chaitali <ch...@freestoneinfotech.com> on 2020/08/11 15:16:16 UTC
Re: Review Request 72756: ATLAS-3918 : Regression : Renaming table has
exceptions in Atlas application logs
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72756/
-----------------------------------------------------------
(Updated Aug. 11, 2020, 3:16 p.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Bugs: ATLAS-3918
https://issues.apache.org/jira/browse/ATLAS-3918
Repository: atlas
Description (updated)
-------
Issue : Messages are being sent for DDL operations one from HMS and HiveServer2(hive_on_tez) which causes duplicate messages hence it throws exception.
message {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgCreationTime":1594802485264,"message":{"type":"ENTITY_PARTIAL_UPDATE_V2","user":"hrt_qa","entityId":{"typeName":"hive_table","uniqueAttributes":{"qualifiedName":"default.t1@cm"}},"entity":{"entity":{"typeName":"hive_table","attributes":{"owner":"hrt_qa","tableType":"MANAGED_TABLE","temporary":false,"lastAccessTime":1594802469000,"aliases":["t1"],"createTime":1594802469000,"qualifiedName":"default.t2@cm","name":"t2","comment":null,"parameters":{"last_modified_time":"1594802481","rawDataSize":"0","numRows":"0","transient_lastDdlTime":"1594802481","bucketing_version":"2","last_modified_by":"hrt_qa","numFilesErasureCoded":"0","totalSize":"0","transactional_properties":"default","COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"id\":\"true\"}}","numFiles":"0","transactional":"true"},"retention":0},"guid":"-21782793831367970","isIncomplete":false
,"provenanceType":0,"version":0,"proxy":false}}}}
This patch fix it by providing a property in configuration when enabled the HiveServer2 will send ddl messages and if disabled then it will only send non ddl messages.
Diffs
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 651323490
Diff: https://reviews.apache.org/r/72756/diff/3/
Testing (updated)
-------
Tested by creating and renaming the hive_table and it doesn't throw above error message when the property is set to false :
> create table t1(id int);
> alter table t1 rename to t2;
Thanks,
chaitali
Re: Review Request 72756: ATLAS-3918 : Regression : Renaming table has
exceptions in Atlas application logs
Posted by chaitali <ch...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72756/
-----------------------------------------------------------
(Updated Sept. 14, 2020, 11:36 a.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Bugs: ATLAS-3918
https://issues.apache.org/jira/browse/ATLAS-3918
Repository: atlas
Description
-------
Issue : Messages are being sent for DDL operations one from HMS and HiveServer2 which causes duplicate messages hence it throws exception:
message {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgCreationTime":1594802485264,"message":{"type":"ENTITY_PARTIAL_UPDATE_V2","user":"hrt_qa","entityId":{"typeName":"hive_table","uniqueAttributes":{"qualifiedName":"default.t1@cm"}},"entity":{"entity":{"typeName":"hive_table","attributes":{"owner":"hrt_qa","tableType":"MANAGED_TABLE","temporary":false,"lastAccessTime":1594802469000,"aliases":["t1"],"createTime":1594802469000,"qualifiedName":"default.t2@cm","name":"t2","comment":null,"parameters":{"last_modified_time":"1594802481","rawDataSize":"0","numRows":"0","transient_lastDdlTime":"1594802481","bucketing_version":"2","last_modified_by":"hrt_qa","numFilesErasureCoded":"0","totalSize":"0","transactional_properties":"default","COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"id\":\"true\"}}","numFiles":"0","transactional":"true"},"retention":0},"guid":"-21782793831367970","isIncomplete":false
,"provenanceType":0,"version":0,"proxy":false}}}}
This patch fix it by providing a property "DDL_MESSAGES_ENABLE" in configuration when enabled the HiveServer2 will send ddl messages and if disabled then it will only send non ddl messages.
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java 6961fa7c2
intg/src/main/java/org/apache/atlas/model/notification/HookNotification.java 5b5fa04e2
intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java f02c05fff
Diff: https://reviews.apache.org/r/72756/diff/4/
Changes: https://reviews.apache.org/r/72756/diff/3-4/
Testing
-------
Tested by creating and renaming the hive_table hence ddl messages are not sent from hiveserver2 after the property is set to false and so no exception is thrown:
> create table t1(id int);
> alter table t1 rename to t2;
Thanks,
chaitali
Re: Review Request 72756: ATLAS-3918 : Regression : Renaming table has
exceptions in Atlas application logs
Posted by chaitali <ch...@freestoneinfotech.com>.
> On Aug. 12, 2020, 1:45 a.m., Madhan Neethiraj wrote:
> > For some DDLs like 'create view', HMS doesn't have all details. So, it is critical that we process DDLs in HiveServer2.
> >
> > Consider adding a new message-type ENTITY_RENAME, and for such messages Atlas should ignore if source entity is not found.
I have updated patch with your suggested changes kindly review.
- chaitali
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72756/#review221551
-----------------------------------------------------------
On Sept. 14, 2020, 11:36 a.m., chaitali wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72756/
> -----------------------------------------------------------
>
> (Updated Sept. 14, 2020, 11:36 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3918
> https://issues.apache.org/jira/browse/ATLAS-3918
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Issue : Messages are being sent for DDL operations one from HMS and HiveServer2 which causes duplicate messages hence it throws exception:
>
> message {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgCreationTime":1594802485264,"message":{"type":"ENTITY_PARTIAL_UPDATE_V2","user":"hrt_qa","entityId":{"typeName":"hive_table","uniqueAttributes":{"qualifiedName":"default.t1@cm"}},"entity":{"entity":{"typeName":"hive_table","attributes":{"owner":"hrt_qa","tableType":"MANAGED_TABLE","temporary":false,"lastAccessTime":1594802469000,"aliases":["t1"],"createTime":1594802469000,"qualifiedName":"default.t2@cm","name":"t2","comment":null,"parameters":{"last_modified_time":"1594802481","rawDataSize":"0","numRows":"0","transient_lastDdlTime":"1594802481","bucketing_version":"2","last_modified_by":"hrt_qa","numFilesErasureCoded":"0","totalSize":"0","transactional_properties":"default","COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"id\":\"true\"}}","numFiles":"0","transactional":"true"},"retention":0},"guid":"-21782793831367970","isIncomplete":fal
se,"provenanceType":0,"version":0,"proxy":false}}}}
>
> This patch fix it by providing a property "DDL_MESSAGES_ENABLE" in configuration when enabled the HiveServer2 will send ddl messages and if disabled then it will only send non ddl messages.
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java 6961fa7c2
> intg/src/main/java/org/apache/atlas/model/notification/HookNotification.java 5b5fa04e2
> intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java f02c05fff
>
>
> Diff: https://reviews.apache.org/r/72756/diff/4/
>
>
> Testing
> -------
>
> Tested by creating and renaming the hive_table hence ddl messages are not sent from hiveserver2 after the property is set to false and so no exception is thrown:
> > create table t1(id int);
>
> > alter table t1 rename to t2;
>
>
> Thanks,
>
> chaitali
>
>
Re: Review Request 72756: ATLAS-3918 : Regression : Renaming table has
exceptions in Atlas application logs
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72756/#review221551
-----------------------------------------------------------
For some DDLs like 'create view', HMS doesn't have all details. So, it is critical that we process DDLs in HiveServer2.
Consider adding a new message-type ENTITY_RENAME, and for such messages Atlas should ignore if source entity is not found.
- Madhan Neethiraj
On Aug. 11, 2020, 3:21 p.m., chaitali wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72756/
> -----------------------------------------------------------
>
> (Updated Aug. 11, 2020, 3:21 p.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3918
> https://issues.apache.org/jira/browse/ATLAS-3918
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Issue : Messages are being sent for DDL operations one from HMS and HiveServer2 which causes duplicate messages hence it throws exception:
>
> message {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgCreationTime":1594802485264,"message":{"type":"ENTITY_PARTIAL_UPDATE_V2","user":"hrt_qa","entityId":{"typeName":"hive_table","uniqueAttributes":{"qualifiedName":"default.t1@cm"}},"entity":{"entity":{"typeName":"hive_table","attributes":{"owner":"hrt_qa","tableType":"MANAGED_TABLE","temporary":false,"lastAccessTime":1594802469000,"aliases":["t1"],"createTime":1594802469000,"qualifiedName":"default.t2@cm","name":"t2","comment":null,"parameters":{"last_modified_time":"1594802481","rawDataSize":"0","numRows":"0","transient_lastDdlTime":"1594802481","bucketing_version":"2","last_modified_by":"hrt_qa","numFilesErasureCoded":"0","totalSize":"0","transactional_properties":"default","COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"id\":\"true\"}}","numFiles":"0","transactional":"true"},"retention":0},"guid":"-21782793831367970","isIncomplete":fal
se,"provenanceType":0,"version":0,"proxy":false}}}}
>
> This patch fix it by providing a property "DDL_MESSAGES_ENABLE" in configuration when enabled the HiveServer2 will send ddl messages and if disabled then it will only send non ddl messages.
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 651323490
>
>
> Diff: https://reviews.apache.org/r/72756/diff/3/
>
>
> Testing
> -------
>
> Tested by creating and renaming the hive_table hence ddl messages are not sent from hiveserver2 after the property is set to false and so no exception is thrown:
> > create table t1(id int);
>
> > alter table t1 rename to t2;
>
>
> Thanks,
>
> chaitali
>
>
Re: Review Request 72756: ATLAS-3918 : Regression : Renaming table has
exceptions in Atlas application logs
Posted by chaitali <ch...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72756/
-----------------------------------------------------------
(Updated Aug. 11, 2020, 3:21 p.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Bugs: ATLAS-3918
https://issues.apache.org/jira/browse/ATLAS-3918
Repository: atlas
Description (updated)
-------
Issue : Messages are being sent for DDL operations one from HMS and HiveServer2 which causes duplicate messages hence it throws exception:
message {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgCreationTime":1594802485264,"message":{"type":"ENTITY_PARTIAL_UPDATE_V2","user":"hrt_qa","entityId":{"typeName":"hive_table","uniqueAttributes":{"qualifiedName":"default.t1@cm"}},"entity":{"entity":{"typeName":"hive_table","attributes":{"owner":"hrt_qa","tableType":"MANAGED_TABLE","temporary":false,"lastAccessTime":1594802469000,"aliases":["t1"],"createTime":1594802469000,"qualifiedName":"default.t2@cm","name":"t2","comment":null,"parameters":{"last_modified_time":"1594802481","rawDataSize":"0","numRows":"0","transient_lastDdlTime":"1594802481","bucketing_version":"2","last_modified_by":"hrt_qa","numFilesErasureCoded":"0","totalSize":"0","transactional_properties":"default","COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"id\":\"true\"}}","numFiles":"0","transactional":"true"},"retention":0},"guid":"-21782793831367970","isIncomplete":false
,"provenanceType":0,"version":0,"proxy":false}}}}
This patch fix it by providing a property "DDL_MESSAGES_ENABLE" in configuration when enabled the HiveServer2 will send ddl messages and if disabled then it will only send non ddl messages.
Diffs
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 651323490
Diff: https://reviews.apache.org/r/72756/diff/3/
Testing (updated)
-------
Tested by creating and renaming the hive_table hence ddl messages are not sent from hiveserver2 after the property is set to false and so no exception is thrown:
> create table t1(id int);
> alter table t1 rename to t2;
Thanks,
chaitali