You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Nixon Rodrigues <ni...@freestoneinfotech.com> on 2017/08/01 11:07:39 UTC
Review Request 61274: ATLAS-1944 - Fix for
ConcurrentModificationException
Exception in HookConsumer (KafkaConsumer) while stopping Atlas server
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61274/
-----------------------------------------------------------
Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
Bugs: ATLAS-1944
https://issues.apache.org/jira/browse/ATLAS-1944
Repository: atlas
Description
-------
Background:- NotificationHookConsumer was throwing ConcurrentModificationException exception while shutting down the consumer thread. The stop method of thread was calling the kafkacosumer.stop method while in run method of HookConsumer which is busy in cosuming the kafka message by polling kafka server. This simaltanous action of stoping and polling causes ConcurrentModificationException.
Fix:-
* Added try finally block in run method which wraps the while loop and when loop ends the finally blocks calls the consumer close method is called which ensures that close is called after consuming is ended.
* Replaced the ShutdownThread interface which was delaying the shutdown process of consumer with Runnable interface used earlier.
Reference : - https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
Diffs
-----
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d431176
notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java 0bd75e1
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java bcee00c
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java b8255b3
Diff: https://reviews.apache.org/r/61274/diff/1/
Testing
-------
Tested the Atlas kafkaconsumer from hive hook and shutdown of atlas server to check if there is issue while shutdown process.
Unit test and integration test are passing.
Thanks,
Nixon Rodrigues
Re: Review Request 61274: ATLAS-1944 - Fix for
ConcurrentModificationException Exception in HookConsumer (KafkaConsumer)
while stopping Atlas server
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
> On Aug. 1, 2017, 4:25 p.m., Apoorv Naik wrote:
> > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
> > Lines 221 (patched)
> > <https://reviews.apache.org/r/61274/diff/1/?file=1786354#file1786354line222>
> >
> > LOG.info => LOG.debug
Apoorv, this is logged one time and can be helpfull to know if consumer has started, so i am keeping it.
- Nixon
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61274/#review181900
-----------------------------------------------------------
On Aug. 1, 2017, 11:07 a.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61274/
> -----------------------------------------------------------
>
> (Updated Aug. 1, 2017, 11:07 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1944
> https://issues.apache.org/jira/browse/ATLAS-1944
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Background:- NotificationHookConsumer was throwing ConcurrentModificationException exception while shutting down the consumer thread. The stop method of thread was calling the kafkacosumer.stop method while in run method of HookConsumer which is busy in cosuming the kafka message by polling kafka server. This simaltanous action of stoping and polling causes ConcurrentModificationException.
>
> Fix:-
>
> * Added try finally block in run method which wraps the while loop and when loop ends the finally blocks calls the consumer close method is called which ensures that close is called after consuming is ended.
>
> * Replaced the ShutdownThread interface which was delaying the shutdown process of consumer with Runnable interface used earlier.
>
>
> Reference : - https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
>
>
> Diffs
> -----
>
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d431176
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java 0bd75e1
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java bcee00c
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java b8255b3
>
>
> Diff: https://reviews.apache.org/r/61274/diff/1/
>
>
> Testing
> -------
>
> Tested the Atlas kafkaconsumer from hive hook and shutdown of atlas server to check if there is issue while shutdown process.
> Unit test and integration test are passing.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 61274: ATLAS-1944 - Fix for
ConcurrentModificationException Exception in HookConsumer (KafkaConsumer)
while stopping Atlas server
Posted by Apoorv Naik <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61274/#review181900
-----------------------------------------------------------
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
Lines 171 (patched)
<https://reviews.apache.org/r/61274/#comment257694>
LOG.info => LOG.debug
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
Lines 221 (patched)
<https://reviews.apache.org/r/61274/#comment257695>
LOG.info => LOG.debug
- Apoorv Naik
On Aug. 1, 2017, 11:07 a.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61274/
> -----------------------------------------------------------
>
> (Updated Aug. 1, 2017, 11:07 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1944
> https://issues.apache.org/jira/browse/ATLAS-1944
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Background:- NotificationHookConsumer was throwing ConcurrentModificationException exception while shutting down the consumer thread. The stop method of thread was calling the kafkacosumer.stop method while in run method of HookConsumer which is busy in cosuming the kafka message by polling kafka server. This simaltanous action of stoping and polling causes ConcurrentModificationException.
>
> Fix:-
>
> * Added try finally block in run method which wraps the while loop and when loop ends the finally blocks calls the consumer close method is called which ensures that close is called after consuming is ended.
>
> * Replaced the ShutdownThread interface which was delaying the shutdown process of consumer with Runnable interface used earlier.
>
>
> Reference : - https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
>
>
> Diffs
> -----
>
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d431176
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java 0bd75e1
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java bcee00c
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java b8255b3
>
>
> Diff: https://reviews.apache.org/r/61274/diff/1/
>
>
> Testing
> -------
>
> Tested the Atlas kafkaconsumer from hive hook and shutdown of atlas server to check if there is issue while shutdown process.
> Unit test and integration test are passing.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 61274: ATLAS-1944 - Fix for
ConcurrentModificationException Exception in HookConsumer (KafkaConsumer)
while stopping Atlas server
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61274/#review182029
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On Aug. 2, 2017, 8:48 a.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61274/
> -----------------------------------------------------------
>
> (Updated Aug. 2, 2017, 8:48 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1944
> https://issues.apache.org/jira/browse/ATLAS-1944
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Background:- NotificationHookConsumer was throwing ConcurrentModificationException exception while shutting down the consumer thread. The stop method of thread was calling the kafkacosumer.stop method while in run method of HookConsumer which is busy in cosuming the kafka message by polling kafka server. This simaltanous action of stoping and polling causes ConcurrentModificationException.
>
> Fix:-
>
> * Added try finally block in run method which wraps the while loop and when loop ends the finally blocks calls the consumer close method is called which ensures that close is called after consuming is ended.
>
> * Replaced the ShutdownThread interface which was delaying the shutdown process of consumer with Runnable interface used earlier.
>
>
> Reference : - https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
>
>
> Diffs
> -----
>
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d431176
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java 0bd75e1
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java bcee00c
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 51276d3
>
>
> Diff: https://reviews.apache.org/r/61274/diff/2/
>
>
> Testing
> -------
>
> Tested the Atlas kafkaconsumer from hive hook and shutdown of atlas server to check if there is issue while shutdown process.
> Unit test and integration test are passing.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 61274: ATLAS-1944 - Fix for
ConcurrentModificationException Exception in HookConsumer (KafkaConsumer)
while stopping Atlas server
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61274/#review182027
-----------------------------------------------------------
In _commit_ needs: _(offset + 1)_
See this: http://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
- Ashutosh Mestry
On Aug. 2, 2017, 8:48 a.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61274/
> -----------------------------------------------------------
>
> (Updated Aug. 2, 2017, 8:48 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1944
> https://issues.apache.org/jira/browse/ATLAS-1944
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Background:- NotificationHookConsumer was throwing ConcurrentModificationException exception while shutting down the consumer thread. The stop method of thread was calling the kafkacosumer.stop method while in run method of HookConsumer which is busy in cosuming the kafka message by polling kafka server. This simaltanous action of stoping and polling causes ConcurrentModificationException.
>
> Fix:-
>
> * Added try finally block in run method which wraps the while loop and when loop ends the finally blocks calls the consumer close method is called which ensures that close is called after consuming is ended.
>
> * Replaced the ShutdownThread interface which was delaying the shutdown process of consumer with Runnable interface used earlier.
>
>
> Reference : - https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
>
>
> Diffs
> -----
>
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d431176
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java 0bd75e1
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java bcee00c
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 51276d3
>
>
> Diff: https://reviews.apache.org/r/61274/diff/2/
>
>
> Testing
> -------
>
> Tested the Atlas kafkaconsumer from hive hook and shutdown of atlas server to check if there is issue while shutdown process.
> Unit test and integration test are passing.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 61274: ATLAS-1944 - Fix for
ConcurrentModificationException Exception in HookConsumer (KafkaConsumer)
while stopping Atlas server
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61274/
-----------------------------------------------------------
(Updated Aug. 2, 2017, 8:48 a.m.)
Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
This patch handles review comments from Apoorv and Ashutosh.
After adding extended HookConsumer from ShutdownableThread and calling consumer.shutdown(), closing of kafkaconsumer was happening fine before timeout of consumer thread.so added back the ShutdownableThread implementation in this patch.
Bugs: ATLAS-1944
https://issues.apache.org/jira/browse/ATLAS-1944
Repository: atlas
Description
-------
Background:- NotificationHookConsumer was throwing ConcurrentModificationException exception while shutting down the consumer thread. The stop method of thread was calling the kafkacosumer.stop method while in run method of HookConsumer which is busy in cosuming the kafka message by polling kafka server. This simaltanous action of stoping and polling causes ConcurrentModificationException.
Fix:-
* Added try finally block in run method which wraps the while loop and when loop ends the finally blocks calls the consumer close method is called which ensures that close is called after consuming is ended.
* Replaced the ShutdownThread interface which was delaying the shutdown process of consumer with Runnable interface used earlier.
Reference : - https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
Diffs (updated)
-----
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d431176
notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java 0bd75e1
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java bcee00c
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 51276d3
Diff: https://reviews.apache.org/r/61274/diff/2/
Changes: https://reviews.apache.org/r/61274/diff/1-2/
Testing
-------
Tested the Atlas kafkaconsumer from hive hook and shutdown of atlas server to check if there is issue while shutdown process.
Unit test and integration test are passing.
Thanks,
Nixon Rodrigues
Re: Review Request 61274: ATLAS-1944 - Fix for
ConcurrentModificationException Exception in HookConsumer (KafkaConsumer)
while stopping Atlas server
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
> On Aug. 1, 2017, 8:12 p.m., Ashutosh Mestry wrote:
> > Can you please let me know the testing performed to verify this. I am little hesitent because this removes _ShutdownableThread_.
> >
> > See attached my patch that merely calls _shutdown_ on consumer thread. I have done some testing on this and did not see the problems we had seen before.
With calling consumer.stop(); there timeout in thread shutdown with message in log "Timed out waiting for consumer threads to shut down, exiting uncleanly"
But after Adding extended HookConsumer from ShutdownableThread and calling consumer.shutdown(), closing of kafkaconsumer was happening fine before timeout of consumer thread.Now added back the ShutdownableThread implementation.
- Nixon
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61274/#review181916
-----------------------------------------------------------
On Aug. 1, 2017, 11:07 a.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61274/
> -----------------------------------------------------------
>
> (Updated Aug. 1, 2017, 11:07 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1944
> https://issues.apache.org/jira/browse/ATLAS-1944
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Background:- NotificationHookConsumer was throwing ConcurrentModificationException exception while shutting down the consumer thread. The stop method of thread was calling the kafkacosumer.stop method while in run method of HookConsumer which is busy in cosuming the kafka message by polling kafka server. This simaltanous action of stoping and polling causes ConcurrentModificationException.
>
> Fix:-
>
> * Added try finally block in run method which wraps the while loop and when loop ends the finally blocks calls the consumer close method is called which ensures that close is called after consuming is ended.
>
> * Replaced the ShutdownThread interface which was delaying the shutdown process of consumer with Runnable interface used earlier.
>
>
> Reference : - https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
>
>
> Diffs
> -----
>
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d431176
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java 0bd75e1
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java bcee00c
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java b8255b3
>
>
> Diff: https://reviews.apache.org/r/61274/diff/1/
>
>
> Testing
> -------
>
> Tested the Atlas kafkaconsumer from hive hook and shutdown of atlas server to check if there is issue while shutdown process.
> Unit test and integration test are passing.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 61274: ATLAS-1944 - Fix for
ConcurrentModificationException Exception in HookConsumer (KafkaConsumer)
while stopping Atlas server
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61274/#review181916
-----------------------------------------------------------
Can you please let me know the testing performed to verify this. I am little hesitent because this removes _ShutdownableThread_.
See attached my patch that merely calls _shutdown_ on consumer thread. I have done some testing on this and did not see the problems we had seen before.
- Ashutosh Mestry
On Aug. 1, 2017, 11:07 a.m., Nixon Rodrigues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61274/
> -----------------------------------------------------------
>
> (Updated Aug. 1, 2017, 11:07 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-1944
> https://issues.apache.org/jira/browse/ATLAS-1944
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Background:- NotificationHookConsumer was throwing ConcurrentModificationException exception while shutting down the consumer thread. The stop method of thread was calling the kafkacosumer.stop method while in run method of HookConsumer which is busy in cosuming the kafka message by polling kafka server. This simaltanous action of stoping and polling causes ConcurrentModificationException.
>
> Fix:-
>
> * Added try finally block in run method which wraps the while loop and when loop ends the finally blocks calls the consumer close method is called which ensures that close is called after consuming is ended.
>
> * Replaced the ShutdownThread interface which was delaying the shutdown process of consumer with Runnable interface used earlier.
>
>
> Reference : - https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
>
>
> Diffs
> -----
>
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d431176
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java 0bd75e1
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java bcee00c
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java b8255b3
>
>
> Diff: https://reviews.apache.org/r/61274/diff/1/
>
>
> Testing
> -------
>
> Tested the Atlas kafkaconsumer from hive hook and shutdown of atlas server to check if there is issue while shutdown process.
> Unit test and integration test are passing.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>