You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Saqeeb Shaikh <sa...@freestoneinfotech.com> on 2019/07/03 14:52:39 UTC
Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/
-----------------------------------------------------------
Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
Bugs: ATLAS-3313
https://issues.apache.org/jira/browse/ATLAS-3313
Repository: atlas
Description
-------
Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
Add following properties in atlas:
atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
Diffs
-----
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
Diff: https://reviews.apache.org/r/71006/diff/1/
Testing
-------
Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
mvn clean package
Some test cases are failing on my local (will work on fixing the same).
Thanks,
Saqeeb Shaikh
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216360
-----------------------------------------------------------
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 240 (original), 275 (patched)
<https://reviews.apache.org/r/71006/#comment303561>
kafka consumer needs to be checked if its not open already - "!isKafkaConsumerOpen(consumer)". This check is needed because of ATLAS-2751.
- Sarath Subramanian
On July 3, 2019, 7:52 a.m., Saqeeb Shaikh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> -----------------------------------------------------------
>
> (Updated July 3, 2019, 7:52 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
>
> To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
>
>
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
>
>
> Diff: https://reviews.apache.org/r/71006/diff/1/
>
>
> Testing
> -------
>
> Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
>
> mvn clean package
> Some test cases are failing on my local (will work on fixing the same).
>
>
> Thanks,
>
> Saqeeb Shaikh
>
>
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216349
-----------------------------------------------------------
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 187 (original), 221 (patched)
<https://reviews.apache.org/r/71006/#comment303541>
Consider renaming createProducer() as getOrCreateProducer() and replace #218 - #223 with:
KafkaProducer producer = getOrCreateProducer(type);
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 271 (original), 316 (patched)
<https://reviews.apache.org/r/71006/#comment303542>
move 'synchronized' inside, to cover lines #322 - #324
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 272 (original), 317 (patched)
<https://reviews.apache.org/r/71006/#comment303543>
LOG.info => LOG.debug - in #317 and #327.
- Madhan Neethiraj
On July 3, 2019, 2:52 p.m., Saqeeb Shaikh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> -----------------------------------------------------------
>
> (Updated July 3, 2019, 2:52 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
>
> To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
>
>
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
>
>
> Diff: https://reviews.apache.org/r/71006/diff/1/
>
>
> Testing
> -------
>
> Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
>
> mvn clean package
> Some test cases are failing on my local (will work on fixing the same).
>
>
> Thanks,
>
> Saqeeb Shaikh
>
>
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Saqeeb Shaikh <sa...@freestoneinfotech.com>.
> On July 3, 2019, 4:19 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java
> > Lines 324 (patched)
> > <https://reviews.apache.org/r/71006/diff/1/?file=2153481#file2153481line325>
> >
> > Please add test for this.
Working on test cases for AtlasMetricUtil.java
- Saqeeb
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216346
-----------------------------------------------------------
On July 8, 2019, 1:26 p.m., Saqeeb Shaikh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> -----------------------------------------------------------
>
> (Updated July 8, 2019, 1:26 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
>
> To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
>
>
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
> notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 847caa3
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
>
>
> Diff: https://reviews.apache.org/r/71006/diff/2/
>
>
> Testing
> -------
>
> Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
>
> mvn clean package
> Some test cases are failing on my local (will work on fixing the same).
>
>
> Thanks,
>
> Saqeeb Shaikh
>
>
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216346
-----------------------------------------------------------
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 185 (original), 217 (patched)
<https://reviews.apache.org/r/71006/#comment303526>
Rename type to notificationType for clarity.
repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java
Lines 324 (patched)
<https://reviews.apache.org/r/71006/#comment303527>
Please add test for this.
- Ashutosh Mestry
On July 3, 2019, 2:52 p.m., Saqeeb Shaikh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> -----------------------------------------------------------
>
> (Updated July 3, 2019, 2:52 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
>
> To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
>
>
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
>
>
> Diff: https://reviews.apache.org/r/71006/diff/1/
>
>
> Testing
> -------
>
> Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
>
> mvn clean package
> Some test cases are failing on my local (will work on fixing the same).
>
>
> Thanks,
>
> Saqeeb Shaikh
>
>
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Sridhar K <Sr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216439
-----------------------------------------------------------
- Sridhar K
On July 8, 2019, 1:26 p.m., Saqeeb Shaikh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> -----------------------------------------------------------
>
> (Updated July 8, 2019, 1:26 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
>
> To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
>
>
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
> notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 847caa3
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
>
>
> Diff: https://reviews.apache.org/r/71006/diff/2/
>
>
> Testing
> -------
>
> Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
>
> mvn clean package
> Some test cases are failing on my local (will work on fixing the same).
>
>
> Thanks,
>
> Saqeeb Shaikh
>
>
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216520
-----------------------------------------------------------
Ship it!
Ship It!
- Nixon Rodrigues
On July 11, 2019, 8:41 a.m., Saqeeb Shaikh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> -----------------------------------------------------------
>
> (Updated July 11, 2019, 8:41 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
>
> To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
>
>
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
> notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 847caa3
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
>
>
> Diff: https://reviews.apache.org/r/71006/diff/4/
>
>
> Testing
> -------
>
> Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
>
> mvn clean package
> Some test cases are failing on my local (will work on fixing the same).
>
>
> Thanks,
>
> Saqeeb Shaikh
>
>
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216530
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On July 11, 2019, 8:41 a.m., Saqeeb Shaikh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> -----------------------------------------------------------
>
> (Updated July 11, 2019, 8:41 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
>
> To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
>
>
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
> notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 847caa3
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
>
>
> Diff: https://reviews.apache.org/r/71006/diff/4/
>
>
> Testing
> -------
>
> Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
>
> mvn clean package
> Some test cases are failing on my local (will work on fixing the same).
>
>
> Thanks,
>
> Saqeeb Shaikh
>
>
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Saqeeb Shaikh <sa...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/
-----------------------------------------------------------
(Updated July 11, 2019, 8:41 a.m.)
Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updated implementation of KafkaNotification.
Bugs: ATLAS-3313
https://issues.apache.org/jira/browse/ATLAS-3313
Repository: atlas
Description
-------
Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
Add following properties in atlas:
atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 847caa3
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
Diff: https://reviews.apache.org/r/71006/diff/4/
Changes: https://reviews.apache.org/r/71006/diff/3-4/
Testing
-------
Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
mvn clean package
Some test cases are failing on my local (will work on fixing the same).
Thanks,
Saqeeb Shaikh
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Saqeeb Shaikh <sa...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/
-----------------------------------------------------------
(Updated July 10, 2019, 3:22 p.m.)
Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Addressed new review comments.
Bugs: ATLAS-3313
https://issues.apache.org/jira/browse/ATLAS-3313
Repository: atlas
Description
-------
Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
Add following properties in atlas:
atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 847caa3
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
Diff: https://reviews.apache.org/r/71006/diff/3/
Changes: https://reviews.apache.org/r/71006/diff/2-3/
Testing
-------
Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
mvn clean package
Some test cases are failing on my local (will work on fixing the same).
Thanks,
Saqeeb Shaikh
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Sridhar K <Sr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216438
-----------------------------------------------------------
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 91 (patched)
<https://reviews.apache.org/r/71006/#comment303677>
Can we please rename this method?...too generic name.
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 100 (patched)
<https://reviews.apache.org/r/71006/#comment303679>
is it possible to go over all strings and drop strings that have zero length after trimming...It will cover scenarios like the following..
"value1",,"value2",
"value1", ,"value2",
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 103 (patched)
<https://reviews.apache.org/r/71006/#comment303678>
Can we please rename this method?...too generic name.
is it possible to go over all strings and drop strings that have zero length after trimming...It will cover scenarios like the following..
"value1",,"value2",
"value1", ,"value2",
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java
Line 58 (original), 58 (patched)
<https://reviews.apache.org/r/71006/#comment303680>
You are changing the metrics names. Will there be any impact on existing metric data...Question is valid, if these values are persisted some where.
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Lines 176 (patched)
<https://reviews.apache.org/r/71006/#comment303676>
Can you also put a message saying that the number of consumers will be set to number of topics.
I will also put a message in startConsumers() method that is the initial call in hierarchy.
- Sridhar K
On July 8, 2019, 1:26 p.m., Saqeeb Shaikh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> -----------------------------------------------------------
>
> (Updated July 8, 2019, 1:26 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
>
> To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
>
>
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
> notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 847caa3
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
>
>
> Diff: https://reviews.apache.org/r/71006/diff/2/
>
>
> Testing
> -------
>
> Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
>
> mvn clean package
> Some test cases are failing on my local (will work on fixing the same).
>
>
> Thanks,
>
> Saqeeb Shaikh
>
>
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216452
-----------------------------------------------------------
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 91 (patched)
<https://reviews.apache.org/r/71006/#comment303688>
I agree. Let's rename to getStringArray() - to be in sync with method names in org.apache.commons.configuration.Configuration.
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 100 (patched)
<https://reviews.apache.org/r/71006/#comment303689>
I suggest we leave trimming and handling of empty value to the callers. It is possible that an empty string is a valid value for some configurations.
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java
Line 58 (original), 58 (patched)
<https://reviews.apache.org/r/71006/#comment303690>
metrics values are not persisted in Atlas. And given we need to track offsets for multiple topics, the metrics name need to change.
- Madhan Neethiraj
On July 8, 2019, 1:26 p.m., Saqeeb Shaikh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> -----------------------------------------------------------
>
> (Updated July 8, 2019, 1:26 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
>
> To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
>
>
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
> notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 847caa3
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
>
>
> Diff: https://reviews.apache.org/r/71006/diff/2/
>
>
> Testing
> -------
>
> Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
>
> mvn clean package
> Some test cases are failing on my local (will work on fixing the same).
>
>
> Thanks,
>
> Saqeeb Shaikh
>
>
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Saqeeb Shaikh <sa...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/
-----------------------------------------------------------
(Updated July 8, 2019, 1:26 p.m.)
Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Addressed review comments
Bugs: ATLAS-3313
https://issues.apache.org/jira/browse/ATLAS-3313
Repository: atlas
Description
-------
Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
Add following properties in atlas:
atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 847caa3
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
Diff: https://reviews.apache.org/r/71006/diff/2/
Changes: https://reviews.apache.org/r/71006/diff/1-2/
Testing
-------
Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
mvn clean package
Some test cases are failing on my local (will work on fixing the same).
Thanks,
Saqeeb Shaikh
Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process
notifications from multiple Kafka topics
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216345
-----------------------------------------------------------
Can you please add a unit test to KafkaConsumerTest to include this scenario.
Overall impelemnetation looks simple enough. We are still little away for scaling Atlas.
I was expecting that the NotificationHookConsumer will consume a particular topic. But I suppose that will need another set of changes.
- Ashutosh Mestry
On July 3, 2019, 2:52 p.m., Saqeeb Shaikh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> -----------------------------------------------------------
>
> (Updated July 3, 2019, 2:52 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
>
> To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics.
>
>
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
> intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 5c840c3
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java b04aba9
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 449eb6f
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 263903b
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c963830
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java d5a4412
> repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
> repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java b56019e
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 1f8e810
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fb3ff26
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java 3e35511
>
>
> Diff: https://reviews.apache.org/r/71006/diff/1/
>
>
> Testing
> -------
>
> Tested by publishing create hive table JSON using kafka-console-producer on custom topics configured above. I was able to search the entities in atlas. Also tested basic use case by creating database and table from hive console.
>
> mvn clean package
> Some test cases are failing on my local (will work on fixing the same).
>
>
> Thanks,
>
> Saqeeb Shaikh
>
>