You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by chaitali <ch...@freestoneinfotech.com> on 2021/10/04 11:17:46 UTC
Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to
capture hook version & message source
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/
-----------------------------------------------------------
(Updated Oct. 4, 2021, 11:17 a.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
Bugs: ATLAS-4428
https://issues.apache.org/jira/browse/ATLAS-4428
Repository: atlas
Description
-------
Atlas hook messages doesn't specify the source information of the messages coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, HBase, Kafka.
Each hook message already captures, the msgSourceIP, msgCreatedBy and msgCreationTime info. In addition to these, we should capture the msgSource information as well.
When messages come from different hook versions, Atlas server should be aware of the version of different hook messages to process the message based on the hook version.
Diffs (updated)
-----
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 8c09d3394
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 0ab06f2c5
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 3cc7b3bc6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java 6a492c214
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java 10ae08f6d
addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 4785960a7
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 779c5cb20
common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311
intg/pom.xml 691cdf0a3
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java ff45d5713
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java 5869910cb
intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java PRE-CREATION
notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java c45a1da95
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 3d8d9cc0a
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 0c92c300e
notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 8078a6ceb
notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java 13eafb6b1
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
Diff: https://reviews.apache.org/r/73595/diff/4/
Changes: https://reviews.apache.org/r/73595/diff/3-4/
Testing
-------
mvn clean install
mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
testcases running successfully
Thanks,
chaitali
Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to
capture hook version & message source
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223575
-----------------------------------------------------------
intg/pom.xml
Lines 149 (patched)
<https://reviews.apache.org/r/73595/#comment312647>
Remove this and instead add atlas-buildinfo.properties here:
build.epoch=${timestamp}
project.version=${pom.version}
build.version=${pom.version}
vc.revision=${buildNumber}
This will avoid copying as build step and will keep the build version true to the project.
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
Line 139 (original), 141 (patched)
<https://reviews.apache.org/r/73595/#comment312648>
Add assertEquals verifying contents of MessageSource.
- Ashutosh Mestry
On Oct. 4, 2021, 3:59 p.m., chaitali wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> -----------------------------------------------------------
>
> (Updated Oct. 4, 2021, 3:59 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas hook messages doesn't specify the source information of the messages coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, HBase, Kafka.
>
> Each hook message already captures, the msgSourceIP, msgCreatedBy and msgCreationTime info. In addition to these, we should capture the msgSource information as well.
>
> When messages come from different hook versions, Atlas server should be aware of the version of different hook messages to process the message based on the hook version.
>
>
> Diffs
> -----
>
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 8c09d3394
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 0ab06f2c5
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 3cc7b3bc6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java 6a492c214
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java 10ae08f6d
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 4785960a7
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 779c5cb20
> common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311
> intg/pom.xml 691cdf0a3
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java ff45d5713
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java 5869910cb
> intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java c45a1da95
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 3d8d9cc0a
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 0c92c300e
> notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 8078a6ceb
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java 13eafb6b1
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
>
>
> Diff: https://reviews.apache.org/r/73595/diff/6/
>
>
> Testing
> -------
>
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
>
>
> Thanks,
>
> chaitali
>
>
Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to
capture hook version & message source
Posted by chaitali <ch...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/
-----------------------------------------------------------
(Updated Oct. 29, 2021, 8:30 a.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
Bugs: ATLAS-4428
https://issues.apache.org/jira/browse/ATLAS-4428
Repository: atlas
Description
-------
Atlas hook messages doesn't specify the source information of the messages coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, HBase, Kafka.
Each hook message already captures, the msgSourceIP, msgCreatedBy and msgCreationTime info. In addition to these, we should capture the msgSource information as well.
When messages come from different hook versions, Atlas server should be aware of the version of different hook messages to process the message based on the hook version.
Diffs (updated)
-----
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 8c09d3394
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 0ab06f2c5
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 3cc7b3bc6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java 6a492c214
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java 10ae08f6d
addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 4785960a7
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 779c5cb20
common/src/main/java/org/apache/atlas/repository/Constants.java 2669c8a7e
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java ff45d5713
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java 5869910cb
intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java PRE-CREATION
intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION
notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java c45a1da95
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 3d8d9cc0a
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 0c92c300e
notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 8078a6ceb
notification/src/test/java/org/apache/atlas/notification/AtlasNotificationMessageTest.java 91a195d4b
notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java 13eafb6b1
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
Diff: https://reviews.apache.org/r/73595/diff/10/
Changes: https://reviews.apache.org/r/73595/diff/9-10/
Testing
-------
mvn clean install
mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
testcases running successfully
Thanks,
chaitali
Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to
capture hook version & message source
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223610
-----------------------------------------------------------
Ship it!
Ship It!
- Sarath Subramanian
On Oct. 12, 2021, 11:20 a.m., chaitali wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> -----------------------------------------------------------
>
> (Updated Oct. 12, 2021, 11:20 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas hook messages doesn't specify the source information of the messages coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, HBase, Kafka.
>
> Each hook message already captures, the msgSourceIP, msgCreatedBy and msgCreationTime info. In addition to these, we should capture the msgSource information as well.
>
> When messages come from different hook versions, Atlas server should be aware of the version of different hook messages to process the message based on the hook version.
>
>
> Diffs
> -----
>
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 8c09d3394
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 0ab06f2c5
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 3cc7b3bc6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java 6a492c214
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java 10ae08f6d
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 4785960a7
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 779c5cb20
> common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java ff45d5713
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java 5869910cb
> intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java PRE-CREATION
> intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION
> notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java c45a1da95
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 3d8d9cc0a
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 0c92c300e
> notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 8078a6ceb
> notification/src/test/java/org/apache/atlas/notification/AtlasNotificationMessageTest.java 91a195d4b
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java 13eafb6b1
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
>
>
> Diff: https://reviews.apache.org/r/73595/diff/9/
>
>
> Testing
> -------
>
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
>
>
> Thanks,
>
> chaitali
>
>
Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to
capture hook version & message source
Posted by chaitali <ch...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/
-----------------------------------------------------------
(Updated Oct. 12, 2021, 6:20 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
Bugs: ATLAS-4428
https://issues.apache.org/jira/browse/ATLAS-4428
Repository: atlas
Description
-------
Atlas hook messages doesn't specify the source information of the messages coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, HBase, Kafka.
Each hook message already captures, the msgSourceIP, msgCreatedBy and msgCreationTime info. In addition to these, we should capture the msgSource information as well.
When messages come from different hook versions, Atlas server should be aware of the version of different hook messages to process the message based on the hook version.
Diffs (updated)
-----
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 8c09d3394
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 0ab06f2c5
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 3cc7b3bc6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java 6a492c214
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java 10ae08f6d
addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 4785960a7
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 779c5cb20
common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java ff45d5713
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java 5869910cb
intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java PRE-CREATION
intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION
notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java c45a1da95
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 3d8d9cc0a
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 0c92c300e
notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 8078a6ceb
notification/src/test/java/org/apache/atlas/notification/AtlasNotificationMessageTest.java 91a195d4b
notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java 13eafb6b1
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
Diff: https://reviews.apache.org/r/73595/diff/9/
Changes: https://reviews.apache.org/r/73595/diff/8-9/
Testing
-------
mvn clean install
mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
testcases running successfully
Thanks,
chaitali
Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to
capture hook version & message source
Posted by chaitali <ch...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/
-----------------------------------------------------------
(Updated Oct. 12, 2021, 6:11 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
Bugs: ATLAS-4428
https://issues.apache.org/jira/browse/ATLAS-4428
Repository: atlas
Description
-------
Atlas hook messages doesn't specify the source information of the messages coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, HBase, Kafka.
Each hook message already captures, the msgSourceIP, msgCreatedBy and msgCreationTime info. In addition to these, we should capture the msgSource information as well.
When messages come from different hook versions, Atlas server should be aware of the version of different hook messages to process the message based on the hook version.
Diffs (updated)
-----
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 8c09d3394
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 0ab06f2c5
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 3cc7b3bc6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java 6a492c214
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java 10ae08f6d
addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 4785960a7
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 779c5cb20
common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java ff45d5713
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java 5869910cb
intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java PRE-CREATION
intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION
notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java c45a1da95
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 3d8d9cc0a
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 0c92c300e
notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 8078a6ceb
notification/src/test/java/org/apache/atlas/notification/AtlasNotificationMessageTest.java 91a195d4b
notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java 13eafb6b1
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
Diff: https://reviews.apache.org/r/73595/diff/8/
Changes: https://reviews.apache.org/r/73595/diff/7-8/
Testing
-------
mvn clean install
mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
testcases running successfully
Thanks,
chaitali
Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to
capture hook version & message source
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223578
-----------------------------------------------------------
Fix it, then Ship it!
Ship It!
intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 55 (patched)
<https://reviews.apache.org/r/73595/#comment312657>
Rename hookVersion to version
intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 59 (patched)
<https://reviews.apache.org/r/73595/#comment312654>
Rename: storedHookVersion to storedVersion.
intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 85 (patched)
<https://reviews.apache.org/r/73595/#comment312656>
setSource
intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 87 (patched)
<https://reviews.apache.org/r/73595/#comment312655>
Rename: getVersion
- Ashutosh Mestry
On Oct. 6, 2021, 3:13 p.m., chaitali wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> -----------------------------------------------------------
>
> (Updated Oct. 6, 2021, 3:13 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas hook messages doesn't specify the source information of the messages coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, HBase, Kafka.
>
> Each hook message already captures, the msgSourceIP, msgCreatedBy and msgCreationTime info. In addition to these, we should capture the msgSource information as well.
>
> When messages come from different hook versions, Atlas server should be aware of the version of different hook messages to process the message based on the hook version.
>
>
> Diffs
> -----
>
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 8c09d3394
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 0ab06f2c5
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 3cc7b3bc6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java 6a492c214
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java 10ae08f6d
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 4785960a7
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 779c5cb20
> common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java ff45d5713
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java 5869910cb
> intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java PRE-CREATION
> intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION
> notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java c45a1da95
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 3d8d9cc0a
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 0c92c300e
> notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 8078a6ceb
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java 13eafb6b1
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
>
>
> Diff: https://reviews.apache.org/r/73595/diff/7/
>
>
> Testing
> -------
>
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
>
>
> Thanks,
>
> chaitali
>
>
Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to
capture hook version & message source
Posted by chaitali <ch...@freestoneinfotech.com>.
> On Oct. 7, 2021, 5:11 a.m., Sarath Subramanian wrote:
> > Add tests for cross compatability. If older Atlas server can deserialize the new message structure added by new hook and the other way (older hook sending message to new Atlas server).
I tried adding that test case, but wasn't able to simulate the situation in tests cases where AtlasNotificationBaseMessage POJO didn't have the message source attribute and in the test case the same POJO had the attribute and also the vice-versa. But I have tested this scenario manually by using old version in Hook and updated version in Server and vice-versa
- chaitali
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223579
-----------------------------------------------------------
On Oct. 12, 2021, 6:20 p.m., chaitali wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> -----------------------------------------------------------
>
> (Updated Oct. 12, 2021, 6:20 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas hook messages doesn't specify the source information of the messages coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, HBase, Kafka.
>
> Each hook message already captures, the msgSourceIP, msgCreatedBy and msgCreationTime info. In addition to these, we should capture the msgSource information as well.
>
> When messages come from different hook versions, Atlas server should be aware of the version of different hook messages to process the message based on the hook version.
>
>
> Diffs
> -----
>
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 8c09d3394
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 0ab06f2c5
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 3cc7b3bc6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java 6a492c214
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java 10ae08f6d
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 4785960a7
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 779c5cb20
> common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java ff45d5713
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java 5869910cb
> intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java PRE-CREATION
> intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION
> notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java c45a1da95
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 3d8d9cc0a
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 0c92c300e
> notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 8078a6ceb
> notification/src/test/java/org/apache/atlas/notification/AtlasNotificationMessageTest.java 91a195d4b
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java 13eafb6b1
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
>
>
> Diff: https://reviews.apache.org/r/73595/diff/9/
>
>
> Testing
> -------
>
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
>
>
> Thanks,
>
> chaitali
>
>
Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to
capture hook version & message source
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223579
-----------------------------------------------------------
Fix it, then Ship it!
Add tests for cross compatability. If older Atlas server can deserialize the new message structure added by new hook and the other way (older hook sending message to new Atlas server).
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
Line 56 (original)
<https://reviews.apache.org/r/73595/#comment312658>
nit: revert new line change
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
Line 259 (original)
<https://reviews.apache.org/r/73595/#comment312659>
nit: revert new line change
intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 62 (patched)
<https://reviews.apache.org/r/73595/#comment312660>
move private methods after public methods
- Sarath Subramanian
On Oct. 6, 2021, 8:13 a.m., chaitali wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> -----------------------------------------------------------
>
> (Updated Oct. 6, 2021, 8:13 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas hook messages doesn't specify the source information of the messages coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, HBase, Kafka.
>
> Each hook message already captures, the msgSourceIP, msgCreatedBy and msgCreationTime info. In addition to these, we should capture the msgSource information as well.
>
> When messages come from different hook versions, Atlas server should be aware of the version of different hook messages to process the message based on the hook version.
>
>
> Diffs
> -----
>
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 8c09d3394
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 0ab06f2c5
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 3cc7b3bc6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java 6a492c214
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java 10ae08f6d
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 4785960a7
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 779c5cb20
> common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java ff45d5713
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java 5869910cb
> intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java PRE-CREATION
> intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION
> notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java c45a1da95
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 3d8d9cc0a
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 0c92c300e
> notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 8078a6ceb
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java 13eafb6b1
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
>
>
> Diff: https://reviews.apache.org/r/73595/diff/7/
>
>
> Testing
> -------
>
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
>
>
> Thanks,
>
> chaitali
>
>
Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to
capture hook version & message source
Posted by chaitali <ch...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/
-----------------------------------------------------------
(Updated Oct. 6, 2021, 3:13 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
Bugs: ATLAS-4428
https://issues.apache.org/jira/browse/ATLAS-4428
Repository: atlas
Description
-------
Atlas hook messages doesn't specify the source information of the messages coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, HBase, Kafka.
Each hook message already captures, the msgSourceIP, msgCreatedBy and msgCreationTime info. In addition to these, we should capture the msgSource information as well.
When messages come from different hook versions, Atlas server should be aware of the version of different hook messages to process the message based on the hook version.
Diffs (updated)
-----
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 8c09d3394
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 0ab06f2c5
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 3cc7b3bc6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java 6a492c214
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java 10ae08f6d
addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 4785960a7
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 779c5cb20
common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java ff45d5713
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java 5869910cb
intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java PRE-CREATION
intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION
notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java c45a1da95
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 3d8d9cc0a
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 0c92c300e
notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 8078a6ceb
notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java 13eafb6b1
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
Diff: https://reviews.apache.org/r/73595/diff/7/
Changes: https://reviews.apache.org/r/73595/diff/6-7/
Testing
-------
mvn clean install
mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
testcases running successfully
Thanks,
chaitali
Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to
capture hook version & message source
Posted by chaitali <ch...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/
-----------------------------------------------------------
(Updated Oct. 4, 2021, 3:59 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
Bugs: ATLAS-4428
https://issues.apache.org/jira/browse/ATLAS-4428
Repository: atlas
Description
-------
Atlas hook messages doesn't specify the source information of the messages coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, HBase, Kafka.
Each hook message already captures, the msgSourceIP, msgCreatedBy and msgCreationTime info. In addition to these, we should capture the msgSource information as well.
When messages come from different hook versions, Atlas server should be aware of the version of different hook messages to process the message based on the hook version.
Diffs (updated)
-----
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 8c09d3394
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 0ab06f2c5
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 3cc7b3bc6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java 6a492c214
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java 10ae08f6d
addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 4785960a7
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 779c5cb20
common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311
intg/pom.xml 691cdf0a3
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java ff45d5713
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java 5869910cb
intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java PRE-CREATION
notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java c45a1da95
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 3d8d9cc0a
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 0c92c300e
notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 8078a6ceb
notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java 13eafb6b1
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
Diff: https://reviews.apache.org/r/73595/diff/6/
Changes: https://reviews.apache.org/r/73595/diff/5-6/
Testing
-------
mvn clean install
mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
testcases running successfully
Thanks,
chaitali
Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to
capture hook version & message source
Posted by chaitali <ch...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/
-----------------------------------------------------------
(Updated Oct. 4, 2021, 3:44 p.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
Bugs: ATLAS-4428
https://issues.apache.org/jira/browse/ATLAS-4428
Repository: atlas
Description
-------
Atlas hook messages doesn't specify the source information of the messages coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, HBase, Kafka.
Each hook message already captures, the msgSourceIP, msgCreatedBy and msgCreationTime info. In addition to these, we should capture the msgSource information as well.
When messages come from different hook versions, Atlas server should be aware of the version of different hook messages to process the message based on the hook version.
Diffs (updated)
-----
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 8c09d3394
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 0ab06f2c5
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 3cc7b3bc6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java 6a492c214
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java 10ae08f6d
addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 4785960a7
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 779c5cb20
common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311
intg/pom.xml 691cdf0a3
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java ff45d5713
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java 5869910cb
intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java PRE-CREATION
notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java c45a1da95
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 3d8d9cc0a
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 0c92c300e
notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 8078a6ceb
notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java 13eafb6b1
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
Diff: https://reviews.apache.org/r/73595/diff/5/
Changes: https://reviews.apache.org/r/73595/diff/4-5/
Testing
-------
mvn clean install
mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
testcases running successfully
Thanks,
chaitali