You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by chaitali <ch...@freestoneinfotech.com> on 2020/09/14 11:36:56 UTC
Re: Review Request 72756: ATLAS-3918 : Regression : Renaming table has
exceptions in Atlas application logs
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72756/
-----------------------------------------------------------
(Updated Sept. 14, 2020, 11:36 a.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Bugs: ATLAS-3918
https://issues.apache.org/jira/browse/ATLAS-3918
Repository: atlas
Description
-------
Issue : Messages are being sent for DDL operations one from HMS and HiveServer2 which causes duplicate messages hence it throws exception:
message {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgCreationTime":1594802485264,"message":{"type":"ENTITY_PARTIAL_UPDATE_V2","user":"hrt_qa","entityId":{"typeName":"hive_table","uniqueAttributes":{"qualifiedName":"default.t1@cm"}},"entity":{"entity":{"typeName":"hive_table","attributes":{"owner":"hrt_qa","tableType":"MANAGED_TABLE","temporary":false,"lastAccessTime":1594802469000,"aliases":["t1"],"createTime":1594802469000,"qualifiedName":"default.t2@cm","name":"t2","comment":null,"parameters":{"last_modified_time":"1594802481","rawDataSize":"0","numRows":"0","transient_lastDdlTime":"1594802481","bucketing_version":"2","last_modified_by":"hrt_qa","numFilesErasureCoded":"0","totalSize":"0","transactional_properties":"default","COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"id\":\"true\"}}","numFiles":"0","transactional":"true"},"retention":0},"guid":"-21782793831367970","isIncomplete":false
,"provenanceType":0,"version":0,"proxy":false}}}}
This patch fix it by providing a property "DDL_MESSAGES_ENABLE" in configuration when enabled the HiveServer2 will send ddl messages and if disabled then it will only send non ddl messages.
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java 6961fa7c2
intg/src/main/java/org/apache/atlas/model/notification/HookNotification.java 5b5fa04e2
intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java f02c05fff
Diff: https://reviews.apache.org/r/72756/diff/4/
Changes: https://reviews.apache.org/r/72756/diff/3-4/
Testing
-------
Tested by creating and renaming the hive_table hence ddl messages are not sent from hiveserver2 after the property is set to false and so no exception is thrown:
> create table t1(id int);
> alter table t1 rename to t2;
Thanks,
chaitali