You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Nixon Rodrigues <ni...@freestoneinfotech.com> on 2017/06/05 08:30:46 UTC
Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL protocol
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/
-----------------------------------------------------------
Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
Repository: atlas
Description
-------
This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
Diffs
-----
distro/src/conf/atlas-application.properties 5e59528
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fd
notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 9585827
notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de
notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb10
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d
Diff: https://reviews.apache.org/r/59799/diff/1/
Testing
-------
Tested Kafka consumer on local embeded kafka /zookeeper env.
Tested Atlas Hive hook notifications.
Excecute Quick Start script.
Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
Thanks,
Nixon Rodrigues
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/#review177096
-----------------------------------------------------------
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
Lines 70 (patched)
<https://reviews.apache.org/r/59799/#comment250644>
disable auto commit in config and commit manually here
- Suma Shivaprasad
On June 5, 2017, 8:30 a.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59799/
> -----------------------------------------------------------
>
> (Updated June 5, 2017, 8:30 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 5e59528
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fd
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 9585827
> notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb10
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d
>
>
> Diff: https://reviews.apache.org/r/59799/diff/1/
>
>
> Testing
> -------
>
> Tested Kafka consumer on local embeded kafka /zookeeper env.
> Tested Atlas Hive hook notifications.
> Excecute Quick Start script.
>
> Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
> On June 7, 2017, 6:56 p.m., Suma Shivaprasad wrote:
> > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
> > Line 129 (original), 133 (patched)
> > <https://reviews.apache.org/r/59799/diff/1/?file=1741124#file1741124line138>
> >
> > any reason this was removed?
With this property kafkaconsumer fails connect and ends up with exception, tested this on local embedded kafka, need to test on external standalone kafka
- Nixon
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/#review177208
-----------------------------------------------------------
On June 12, 2017, 3:02 p.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59799/
> -----------------------------------------------------------
>
> (Updated June 12, 2017, 3:02 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 5e59528
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fd
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 9585827
> notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb10
> notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java ad7d93e
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 2126be6
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java a810029
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 13f2f0b
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d
> webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java ac3b538
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java 18fd2ee
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java bdb60a2
> webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java b59d3ee
> webapp/src/test/java/org/apache/atlas/web/integration/EntityJerseyResourceIT.java 310b2e3
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 98a7abc
>
>
> Diff: https://reviews.apache.org/r/59799/diff/2/
>
>
> Testing
> -------
>
> Tested Kafka consumer on local embeded kafka /zookeeper env.
> Tested Atlas Hive hook notifications.
> Excecute Quick Start script.
>
> Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/#review177208
-----------------------------------------------------------
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 129 (original), 133 (patched)
<https://reviews.apache.org/r/59799/#comment250755>
any reason this was removed?
- Suma Shivaprasad
On June 5, 2017, 8:30 a.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59799/
> -----------------------------------------------------------
>
> (Updated June 5, 2017, 8:30 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 5e59528
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fd
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 9585827
> notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb10
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d
>
>
> Diff: https://reviews.apache.org/r/59799/diff/1/
>
>
> Testing
> -------
>
> Tested Kafka consumer on local embeded kafka /zookeeper env.
> Tested Atlas Hive hook notifications.
> Excecute Quick Start script.
>
> Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/#review177632
-----------------------------------------------------------
distro/src/conf/atlas-application.properties
Lines 81 (patched)
<https://reviews.apache.org/r/59799/#comment251325>
this property name maybe confused with the one in line 79
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
Lines 51 (patched)
<https://reviews.apache.org/r/59799/#comment251326>
newKafkaConsumer1 ==> newKafkaConsumer
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
Lines 178 (patched)
<https://reviews.apache.org/r/59799/#comment251331>
change to LOG.info message
- Sarath Subramanian
On June 12, 2017, 8:02 a.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59799/
> -----------------------------------------------------------
>
> (Updated June 12, 2017, 8:02 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 5e59528
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fd
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 9585827
> notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb10
> notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java ad7d93e
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 2126be6
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java a810029
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 13f2f0b
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d
> webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java ac3b538
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java 18fd2ee
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java bdb60a2
> webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java b59d3ee
> webapp/src/test/java/org/apache/atlas/web/integration/EntityJerseyResourceIT.java 310b2e3
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 98a7abc
>
>
> Diff: https://reviews.apache.org/r/59799/diff/2/
>
>
> Testing
> -------
>
> Tested Kafka consumer on local embeded kafka /zookeeper env.
> Tested Atlas Hive hook notifications.
> Excecute Quick Start script.
>
> Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/#review178780
-----------------------------------------------------------
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
Lines 51 (patched)
<https://reviews.apache.org/r/59799/#comment252964>
"newKafkaConsumer" ==> "kafkaConsumer"
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
Lines 63 (patched)
<https://reviews.apache.org/r/59799/#comment253027>
"List" ==> "List<T>" - since the return type of this method is "List<T>". This update would require bunch of other updates..which endup with inconsistencies in types returned/expected. Please review.
- Madhan Neethiraj
On June 21, 2017, 1:11 p.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59799/
> -----------------------------------------------------------
>
> (Updated June 21, 2017, 1:11 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 5e59528
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fd
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 9585827
> notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb10
> notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java ad7d93e
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 2126be6
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java a810029
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 13f2f0b
> typesystem/src/test/resources/atlas-application.properties c4ce5ea
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d
> webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java ac3b538
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java 18fd2ee
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java bdb60a2
> webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java b59d3ee
> webapp/src/test/java/org/apache/atlas/web/integration/EntityJerseyResourceIT.java 310b2e3
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 98a7abc
>
>
> Diff: https://reviews.apache.org/r/59799/diff/4/
>
>
> Testing
> -------
>
> Tested Kafka consumer on local embeded kafka /zookeeper env.
> Tested Atlas Hive hook notifications.
> Excecute Quick Start script.
>
> Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/#review178894
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On June 26, 2017, 8:37 a.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59799/
> -----------------------------------------------------------
>
> (Updated June 26, 2017, 8:37 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1766
> https://issues.apache.org/jira/browse/ATLAS-1766
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 5e59528b
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2c
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fde
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 95858278
> notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de9
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb105
> notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java ad7d93e2
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 2126be69
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java a8100290
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 13f2f0ba
> typesystem/src/test/resources/atlas-application.properties c4ce5eaa
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d2
> webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java ac3b5381
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java 18fd2ee2
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java bdb60a2f
> webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java b59d3ee5
> webapp/src/test/java/org/apache/atlas/web/integration/EntityJerseyResourceIT.java 310b2e3e
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 98a7abcf
>
>
> Diff: https://reviews.apache.org/r/59799/diff/5/
>
>
> Testing
> -------
>
> Tested Kafka consumer on local embeded kafka /zookeeper env.
> Tested Atlas Hive hook notifications.
> Excecute Quick Start script.
>
> Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/
-----------------------------------------------------------
(Updated June 26, 2017, 8:37 a.m.)
Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
Bugs: ATLAS-1766
https://issues.apache.org/jira/browse/ATLAS-1766
Repository: atlas
Description
-------
This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
Diffs
-----
distro/src/conf/atlas-application.properties 5e59528b
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2c
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fde
notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 95858278
notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de9
notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb105
notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java ad7d93e2
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 2126be69
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java a8100290
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 13f2f0ba
typesystem/src/test/resources/atlas-application.properties c4ce5eaa
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d2
webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java ac3b5381
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java 18fd2ee2
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java bdb60a2f
webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java b59d3ee5
webapp/src/test/java/org/apache/atlas/web/integration/EntityJerseyResourceIT.java 310b2e3e
webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 98a7abcf
Diff: https://reviews.apache.org/r/59799/diff/5/
Testing
-------
Tested Kafka consumer on local embeded kafka /zookeeper env.
Tested Atlas Hive hook notifications.
Excecute Quick Start script.
Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
Thanks,
Nixon Rodrigues
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/
-----------------------------------------------------------
(Updated June 26, 2017, 8:37 a.m.)
Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
This patch includes changes to handle review comments from Madhan.
* Tested the patch from hive hook and hive entities are getting created in Atlas.
Repository: atlas
Description
-------
This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
Diffs (updated)
-----
distro/src/conf/atlas-application.properties 5e59528b
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2c
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fde
notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 95858278
notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de9
notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb105
notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java ad7d93e2
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 2126be69
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java a8100290
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 13f2f0ba
typesystem/src/test/resources/atlas-application.properties c4ce5eaa
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d2
webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java ac3b5381
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java 18fd2ee2
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java bdb60a2f
webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java b59d3ee5
webapp/src/test/java/org/apache/atlas/web/integration/EntityJerseyResourceIT.java 310b2e3e
webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 98a7abcf
Diff: https://reviews.apache.org/r/59799/diff/5/
Changes: https://reviews.apache.org/r/59799/diff/4-5/
Testing
-------
Tested Kafka consumer on local embeded kafka /zookeeper env.
Tested Atlas Hive hook notifications.
Excecute Quick Start script.
Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
Thanks,
Nixon Rodrigues
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/
-----------------------------------------------------------
(Updated June 21, 2017, 1:11 p.m.)
Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
This is patch handles review comments and fix for unit test and integration cases.
Tested the consumer from ATLAS_HOOK topics from hive db after creating hive tables.
Repository: atlas
Description
-------
This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
Diffs (updated)
-----
distro/src/conf/atlas-application.properties 5e59528
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fd
notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 9585827
notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de
notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb10
notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java ad7d93e
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 2126be6
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java a810029
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 13f2f0b
typesystem/src/test/resources/atlas-application.properties c4ce5ea
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d
webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java ac3b538
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java 18fd2ee
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java bdb60a2
webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java b59d3ee
webapp/src/test/java/org/apache/atlas/web/integration/EntityJerseyResourceIT.java 310b2e3
webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 98a7abc
Diff: https://reviews.apache.org/r/59799/diff/4/
Changes: https://reviews.apache.org/r/59799/diff/3-4/
Testing
-------
Tested Kafka consumer on local embeded kafka /zookeeper env.
Tested Atlas Hive hook notifications.
Excecute Quick Start script.
Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
Thanks,
Nixon Rodrigues
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/
-----------------------------------------------------------
(Updated June 14, 2017, 3:19 p.m.)
Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
This patch handles review comments from Sarath and testcases fix.
# 2 test cases related to offset autocommit false are failing are commented.
Repository: atlas
Description
-------
This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
Diffs (updated)
-----
distro/src/conf/atlas-application.properties 5e59528
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fd
notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 9585827
notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de
notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb10
notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java ad7d93e
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 2126be6
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java a810029
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 13f2f0b
typesystem/src/test/resources/atlas-application.properties c4ce5ea
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d
webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java ac3b538
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java 18fd2ee
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java bdb60a2
webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java b59d3ee
webapp/src/test/java/org/apache/atlas/web/integration/EntityJerseyResourceIT.java 310b2e3
webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 98a7abc
Diff: https://reviews.apache.org/r/59799/diff/3/
Changes: https://reviews.apache.org/r/59799/diff/2-3/
Testing
-------
Tested Kafka consumer on local embeded kafka /zookeeper env.
Tested Atlas Hive hook notifications.
Excecute Quick Start script.
Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
Thanks,
Nixon Rodrigues
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/
-----------------------------------------------------------
(Updated June 12, 2017, 3:02 p.m.)
Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
This patch includes review comments by Apoorv & Suma. Also includes testcase changes for new-consumer api.
Testcase related auto.commit offset and few others are pending and commented.
Repository: atlas
Description
-------
This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
Diffs (updated)
-----
distro/src/conf/atlas-application.properties 5e59528
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fd
notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 9585827
notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de
notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb10
notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java ad7d93e
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 2126be6
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java a810029
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 13f2f0b
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d
webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java ac3b538
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java 18fd2ee
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java bdb60a2
webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java b59d3ee
webapp/src/test/java/org/apache/atlas/web/integration/EntityJerseyResourceIT.java 310b2e3
webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 98a7abc
Diff: https://reviews.apache.org/r/59799/diff/2/
Changes: https://reviews.apache.org/r/59799/diff/1-2/
Testing
-------
Tested Kafka consumer on local embeded kafka /zookeeper env.
Tested Atlas Hive hook notifications.
Excecute Quick Start script.
Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
Thanks,
Nixon Rodrigues
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
> On June 6, 2017, 8:50 p.m., Suma Shivaprasad wrote:
> > distro/src/conf/atlas-application.properties
> > Lines 81 (patched)
> > <https://reviews.apache.org/r/59799/diff/1/?file=1741121#file1741121line81>
> >
> > why is auto commit set to true. This will cause issues when there is a transient failure on Atlas server and no retries will be attempted
Have added code to commit the record offset.
- Nixon
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/#review177095
-----------------------------------------------------------
On June 21, 2017, 1:11 p.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59799/
> -----------------------------------------------------------
>
> (Updated June 21, 2017, 1:11 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 5e59528
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fd
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 9585827
> notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb10
> notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java ad7d93e
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java 2126be6
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java a810029
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 13f2f0b
> typesystem/src/test/resources/atlas-application.properties c4ce5ea
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d
> webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java ac3b538
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java 18fd2ee
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java bdb60a2
> webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java b59d3ee
> webapp/src/test/java/org/apache/atlas/web/integration/EntityJerseyResourceIT.java 310b2e3
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 98a7abc
>
>
> Diff: https://reviews.apache.org/r/59799/diff/4/
>
>
> Testing
> -------
>
> Tested Kafka consumer on local embeded kafka /zookeeper env.
> Tested Atlas Hive hook notifications.
> Excecute Quick Start script.
>
> Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/#review177095
-----------------------------------------------------------
distro/src/conf/atlas-application.properties
Lines 81 (patched)
<https://reviews.apache.org/r/59799/#comment250643>
why is auto commit set to true. This will cause issues when there is a transient failure on Atlas server and no retries will be attempted
- Suma Shivaprasad
On June 5, 2017, 8:30 a.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59799/
> -----------------------------------------------------------
>
> (Updated June 5, 2017, 8:30 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 5e59528
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fd
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 9585827
> notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb10
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d
>
>
> Diff: https://reviews.apache.org/r/59799/diff/1/
>
>
> Testing
> -------
>
> Tested Kafka consumer on local embeded kafka /zookeeper env.
> Tested Atlas Hive hook notifications.
> Excecute Quick Start script.
>
> Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 59799: ATLAS-1766 : Atlas with Kafka SASL_SSL
protocol
Posted by Apoorv Naik <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59799/#review176920
-----------------------------------------------------------
Overall functionality wise it looks good.
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
Lines 45 (patched)
<https://reviews.apache.org/r/59799/#comment250474>
Change from fully qualified name to just class name.
newKafkaConsumer -> kafkaConsumer / consumer
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
Lines 59 (patched)
<https://reviews.apache.org/r/59799/#comment250475>
Put this under a debug statement.
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
Lines 63 (patched)
<https://reviews.apache.org/r/59799/#comment250476>
Should this be logged to the error log in order to keep the main application log clean (less stacktraces)
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 59 (original), 68 (patched)
<https://reviews.apache.org/r/59799/#comment250477>
Specific imports over wildcard.
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 129 (original), 133 (patched)
<https://reviews.apache.org/r/59799/#comment250478>
Remove commented code
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 259 (original), 241 (patched)
<https://reviews.apache.org/r/59799/#comment250479>
Direct class reference instead of fully qualified one.
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 293 (original), 262 (patched)
<https://reviews.apache.org/r/59799/#comment250480>
StringUtils.isNotEmpty(groupId)
notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java
Lines 40 (patched)
<https://reviews.apache.org/r/59799/#comment250481>
Can you add Javadoc for this ?
- Apoorv Naik
On June 5, 2017, 8:30 a.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59799/
> -----------------------------------------------------------
>
> (Updated June 5, 2017, 8:30 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch include changes that implements kafka new-consumer api for kafkaConsumer for Atlas Notification System.
>
>
> Diffs
> -----
>
> distro/src/conf/atlas-application.properties 5e59528
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 16c0eb2
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 8bd31fd
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java 9585827
> notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java d4d78de
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java a99cb10
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2f8245d
>
>
> Diff: https://reviews.apache.org/r/59799/diff/1/
>
>
> Testing
> -------
>
> Tested Kafka consumer on local embeded kafka /zookeeper env.
> Tested Atlas Hive hook notifications.
> Excecute Quick Start script.
>
> Due to change in kafkaconsumer api the current notification related UT/IT's need to changed, i have not included this change in this patch but will included it shortly.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>