You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by sandeep samudrala <sa...@gmail.com> on 2015/09/07 20:46:40 UTC
Review Request 38166: Add shutdown hook for Falcon
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38166/
-----------------------------------------------------------
Review request for Falcon.
Bugs: FALCON-348
https://issues.apache.org/jira/browse/FALCON-348
Repository: falcon-git
Description
-------
A shutdown hook is missing and services are shutdown in a brute force mode. I'd like to see activemq, titan and such are shutdown gracefully so we don't lose data and see ugly exceptions in the logs.
Diffs
-----
prism/src/main/java/org/apache/falcon/Main.java 96e003c
rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunConsumer.java 9ee94c5
rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunHandler.java f019737
rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java c2cb09e
rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java c6bc36f
rerun/src/main/java/org/apache/falcon/rerun/queue/ActiveMQueue.java 021e4cc
rerun/src/main/java/org/apache/falcon/rerun/service/LateRunService.java 2bb198b
Diff: https://reviews.apache.org/r/38166/diff/
Testing
-------
I have started and stopped the server. Logs look clean and each service gets shutdown properly.
Handled activemq brokerservice and server stops separately.
Thanks,
sandeep samudrala
Re: Review Request 38166: Add shutdown hook for Falcon
Posted by Pallavi Rao <pa...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38166/#review97965
-----------------------------------------------------------
Ship it!
Ship It!
- Pallavi Rao
On Sept. 7, 2015, 6:46 p.m., sandeep samudrala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38166/
> -----------------------------------------------------------
>
> (Updated Sept. 7, 2015, 6:46 p.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-348
> https://issues.apache.org/jira/browse/FALCON-348
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> A shutdown hook is missing and services are shutdown in a brute force mode. I'd like to see activemq, titan and such are shutdown gracefully so we don't lose data and see ugly exceptions in the logs.
>
>
> Diffs
> -----
>
> prism/src/main/java/org/apache/falcon/Main.java 96e003c
> rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunConsumer.java 9ee94c5
> rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunHandler.java f019737
> rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java c2cb09e
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java c6bc36f
> rerun/src/main/java/org/apache/falcon/rerun/queue/ActiveMQueue.java 021e4cc
> rerun/src/main/java/org/apache/falcon/rerun/service/LateRunService.java 2bb198b
>
> Diff: https://reviews.apache.org/r/38166/diff/
>
>
> Testing
> -------
>
> I have started and stopped the server. Logs look clean and each service gets shutdown properly.
> Handled activemq brokerservice and server stops separately.
>
>
> Thanks,
>
> sandeep samudrala
>
>
Re: Review Request 38166: Add shutdown hook for Falcon
Posted by sandeep samudrala <sa...@gmail.com>.
> On Sept. 9, 2015, 7:46 a.m., Ajay Yadava wrote:
> > rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunConsumer.java, line 54
> > <https://reviews.apache.org/r/38166/diff/2/?file=1065875#file1065875line54>
> >
> > Why not use a volatile boolean and use that for shutting down the thread cleanly instead of interrupting it and then doing all the error handling?
Discussed offline, this patch too works, the changes as mentioned above would touch the code base further and testing and is bit convoluted too.
- sandeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38166/#review98156
-----------------------------------------------------------
On Sept. 9, 2015, 6:23 a.m., sandeep samudrala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38166/
> -----------------------------------------------------------
>
> (Updated Sept. 9, 2015, 6:23 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-348
> https://issues.apache.org/jira/browse/FALCON-348
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> A shutdown hook is missing and services are shutdown in a brute force mode. I'd like to see activemq, titan and such are shutdown gracefully so we don't lose data and see ugly exceptions in the logs.
>
>
> Diffs
> -----
>
> prism/src/main/java/org/apache/falcon/Main.java 96e003c
> rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunConsumer.java 9ee94c5
> rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunHandler.java f019737
> rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java c2cb09e
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java c6bc36f
> rerun/src/main/java/org/apache/falcon/rerun/queue/ActiveMQueue.java 021e4cc
> rerun/src/main/java/org/apache/falcon/rerun/service/LateRunService.java 2bb198b
>
> Diff: https://reviews.apache.org/r/38166/diff/
>
>
> Testing
> -------
>
> I have started and stopped the server. Logs look clean and each service gets shutdown properly.
> Handled activemq brokerservice and server stops separately.
>
> Below is how the logs are looking after the patch.
>
> 2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ calling shutdown hook (Main:68)
> 2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ Stopped SocketConnector@0.0.0.0:15000 (log:67)
> 2015-09-07 18:08:02,922 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:65)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:65)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Closing topicSubscriber on topic : FALCON.ENTITY.TOPIC (JMSMessageConsumer:149)
> 2015-09-07 18:08:02,943 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:65)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:58)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:58)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ RetryHandler thread destroyed (RetryService:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:58)
> 2015-09-07 18:08:02,963 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
> 2015-09-07 18:08:02,964 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
> 2015-09-07 18:08:02,965 INFO - [LaterunHandler:] ~ Rerun handler daemon has been interrupted (AbstractRerunConsumer:62)
> 2015-09-07 18:08:02,966 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
> 2015-09-07 18:08:02,967 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,978 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
> 2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
> 2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ LateRun thread destroyed (LateRunService:70)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:65)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:58)
> 2015-09-07 18:08:02,985 INFO - [Thread-0:] ~ Shutting down graph db (MetadataMappingService:202)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:65)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.LogCleanupService (ServiceInitializer:58)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Falcon log cleanup service destroyed (LogCleanupService:80)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.LogCleanupService (ServiceInitializer:65)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~
> ############################################
> Falcon Server (SHUTDOWN)
> ############################################ (ContextStartupListener:94)
> 2015-09-07 18:08:03,047 INFO - [Thread-0:] ~ ActiveMQ Message Broker (localhost, ID:groot-54663-1441649273016-0:1) is shutting down (BrokerService:560)
> 2015-09-07 18:08:03,048 INFO - [Thread-0:] ~ Connector vm://localhost Stopped (TransportConnector:288)
> 2015-09-07 18:08:03,632 INFO - [Thread-0:] ~ Connector tcp://groot:61616 Stopped (TransportConnector:288)
> 2015-09-07 18:08:03,641 INFO - [Thread-0:] ~ JobSchedulerStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/scheduler stopped (JobSchedulerStore:286)
> 2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ PListStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/tmp_storage stopped (PListStore:307)
> 2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ Stopping async queue tasks (KahaDBStore:206)
> 2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopping async topic tasks (KahaDBStore:220)
> 2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopped KahaDB (KahaDBStore:246)
> 2015-09-07 18:08:04,048 INFO - [Thread-0:] ~ ActiveMQ JMS Message Broker (localhost, ID:groot-54663-1441649273016-0:1) stopped (BrokerService:628)
>
>
> Thanks,
>
> sandeep samudrala
>
>
Re: Review Request 38166: Add shutdown hook for Falcon
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38166/#review98156
-----------------------------------------------------------
rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunConsumer.java (line 54)
<https://reviews.apache.org/r/38166/#comment154440>
Why not use a volatile boolean and use that for shutting down the thread cleanly instead of interrupting it and then doing all the error handling?
rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunConsumer.java (line 61)
<https://reviews.apache.org/r/38166/#comment154439>
It will be better to log the stacktrace as the reason for interrupt might vary.
- Ajay Yadava
On Sept. 9, 2015, 6:23 a.m., sandeep samudrala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38166/
> -----------------------------------------------------------
>
> (Updated Sept. 9, 2015, 6:23 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-348
> https://issues.apache.org/jira/browse/FALCON-348
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> A shutdown hook is missing and services are shutdown in a brute force mode. I'd like to see activemq, titan and such are shutdown gracefully so we don't lose data and see ugly exceptions in the logs.
>
>
> Diffs
> -----
>
> prism/src/main/java/org/apache/falcon/Main.java 96e003c
> rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunConsumer.java 9ee94c5
> rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunHandler.java f019737
> rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java c2cb09e
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java c6bc36f
> rerun/src/main/java/org/apache/falcon/rerun/queue/ActiveMQueue.java 021e4cc
> rerun/src/main/java/org/apache/falcon/rerun/service/LateRunService.java 2bb198b
>
> Diff: https://reviews.apache.org/r/38166/diff/
>
>
> Testing
> -------
>
> I have started and stopped the server. Logs look clean and each service gets shutdown properly.
> Handled activemq brokerservice and server stops separately.
>
> Below is how the logs are looking after the patch.
>
> 2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ calling shutdown hook (Main:68)
> 2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ Stopped SocketConnector@0.0.0.0:15000 (log:67)
> 2015-09-07 18:08:02,922 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:65)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:65)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Closing topicSubscriber on topic : FALCON.ENTITY.TOPIC (JMSMessageConsumer:149)
> 2015-09-07 18:08:02,943 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:65)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:58)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:58)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ RetryHandler thread destroyed (RetryService:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:58)
> 2015-09-07 18:08:02,963 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
> 2015-09-07 18:08:02,964 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
> 2015-09-07 18:08:02,965 INFO - [LaterunHandler:] ~ Rerun handler daemon has been interrupted (AbstractRerunConsumer:62)
> 2015-09-07 18:08:02,966 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
> 2015-09-07 18:08:02,967 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,978 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
> 2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
> 2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ LateRun thread destroyed (LateRunService:70)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:65)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:58)
> 2015-09-07 18:08:02,985 INFO - [Thread-0:] ~ Shutting down graph db (MetadataMappingService:202)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:65)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.LogCleanupService (ServiceInitializer:58)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Falcon log cleanup service destroyed (LogCleanupService:80)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.LogCleanupService (ServiceInitializer:65)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~
> ############################################
> Falcon Server (SHUTDOWN)
> ############################################ (ContextStartupListener:94)
> 2015-09-07 18:08:03,047 INFO - [Thread-0:] ~ ActiveMQ Message Broker (localhost, ID:groot-54663-1441649273016-0:1) is shutting down (BrokerService:560)
> 2015-09-07 18:08:03,048 INFO - [Thread-0:] ~ Connector vm://localhost Stopped (TransportConnector:288)
> 2015-09-07 18:08:03,632 INFO - [Thread-0:] ~ Connector tcp://groot:61616 Stopped (TransportConnector:288)
> 2015-09-07 18:08:03,641 INFO - [Thread-0:] ~ JobSchedulerStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/scheduler stopped (JobSchedulerStore:286)
> 2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ PListStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/tmp_storage stopped (PListStore:307)
> 2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ Stopping async queue tasks (KahaDBStore:206)
> 2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopping async topic tasks (KahaDBStore:220)
> 2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopped KahaDB (KahaDBStore:246)
> 2015-09-07 18:08:04,048 INFO - [Thread-0:] ~ ActiveMQ JMS Message Broker (localhost, ID:groot-54663-1441649273016-0:1) stopped (BrokerService:628)
>
>
> Thanks,
>
> sandeep samudrala
>
>
Re: Review Request 38166: Add shutdown hook for Falcon
Posted by sandeep samudrala <sa...@gmail.com>.
> On Sept. 9, 2015, 7:28 a.m., Ajay Yadava wrote:
> > rerun/src/main/java/org/apache/falcon/rerun/queue/ActiveMQueue.java, line 71
> > <https://reviews.apache.org/r/38166/diff/2/?file=1065879#file1065879line71>
> >
> > Why are log statements being removed?
These log statements are repeated. Its logged as error and also thrown as excpetion which is logged again where it is getting caught.
- sandeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38166/#review98154
-----------------------------------------------------------
On Sept. 9, 2015, 6:23 a.m., sandeep samudrala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38166/
> -----------------------------------------------------------
>
> (Updated Sept. 9, 2015, 6:23 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-348
> https://issues.apache.org/jira/browse/FALCON-348
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> A shutdown hook is missing and services are shutdown in a brute force mode. I'd like to see activemq, titan and such are shutdown gracefully so we don't lose data and see ugly exceptions in the logs.
>
>
> Diffs
> -----
>
> prism/src/main/java/org/apache/falcon/Main.java 96e003c
> rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunConsumer.java 9ee94c5
> rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunHandler.java f019737
> rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java c2cb09e
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java c6bc36f
> rerun/src/main/java/org/apache/falcon/rerun/queue/ActiveMQueue.java 021e4cc
> rerun/src/main/java/org/apache/falcon/rerun/service/LateRunService.java 2bb198b
>
> Diff: https://reviews.apache.org/r/38166/diff/
>
>
> Testing
> -------
>
> I have started and stopped the server. Logs look clean and each service gets shutdown properly.
> Handled activemq brokerservice and server stops separately.
>
> Below is how the logs are looking after the patch.
>
> 2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ calling shutdown hook (Main:68)
> 2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ Stopped SocketConnector@0.0.0.0:15000 (log:67)
> 2015-09-07 18:08:02,922 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:65)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:65)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Closing topicSubscriber on topic : FALCON.ENTITY.TOPIC (JMSMessageConsumer:149)
> 2015-09-07 18:08:02,943 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:65)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:58)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:58)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ RetryHandler thread destroyed (RetryService:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:58)
> 2015-09-07 18:08:02,963 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
> 2015-09-07 18:08:02,964 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
> 2015-09-07 18:08:02,965 INFO - [LaterunHandler:] ~ Rerun handler daemon has been interrupted (AbstractRerunConsumer:62)
> 2015-09-07 18:08:02,966 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
> 2015-09-07 18:08:02,967 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,978 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
> 2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
> 2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ LateRun thread destroyed (LateRunService:70)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:65)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:58)
> 2015-09-07 18:08:02,985 INFO - [Thread-0:] ~ Shutting down graph db (MetadataMappingService:202)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:65)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.LogCleanupService (ServiceInitializer:58)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Falcon log cleanup service destroyed (LogCleanupService:80)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.LogCleanupService (ServiceInitializer:65)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~
> ############################################
> Falcon Server (SHUTDOWN)
> ############################################ (ContextStartupListener:94)
> 2015-09-07 18:08:03,047 INFO - [Thread-0:] ~ ActiveMQ Message Broker (localhost, ID:groot-54663-1441649273016-0:1) is shutting down (BrokerService:560)
> 2015-09-07 18:08:03,048 INFO - [Thread-0:] ~ Connector vm://localhost Stopped (TransportConnector:288)
> 2015-09-07 18:08:03,632 INFO - [Thread-0:] ~ Connector tcp://groot:61616 Stopped (TransportConnector:288)
> 2015-09-07 18:08:03,641 INFO - [Thread-0:] ~ JobSchedulerStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/scheduler stopped (JobSchedulerStore:286)
> 2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ PListStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/tmp_storage stopped (PListStore:307)
> 2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ Stopping async queue tasks (KahaDBStore:206)
> 2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopping async topic tasks (KahaDBStore:220)
> 2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopped KahaDB (KahaDBStore:246)
> 2015-09-07 18:08:04,048 INFO - [Thread-0:] ~ ActiveMQ JMS Message Broker (localhost, ID:groot-54663-1441649273016-0:1) stopped (BrokerService:628)
>
>
> Thanks,
>
> sandeep samudrala
>
>
Re: Review Request 38166: Add shutdown hook for Falcon
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38166/#review98154
-----------------------------------------------------------
rerun/src/main/java/org/apache/falcon/rerun/queue/ActiveMQueue.java
<https://reviews.apache.org/r/38166/#comment154433>
Why are log statements being removed?
- Ajay Yadava
On Sept. 9, 2015, 6:23 a.m., sandeep samudrala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38166/
> -----------------------------------------------------------
>
> (Updated Sept. 9, 2015, 6:23 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-348
> https://issues.apache.org/jira/browse/FALCON-348
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> A shutdown hook is missing and services are shutdown in a brute force mode. I'd like to see activemq, titan and such are shutdown gracefully so we don't lose data and see ugly exceptions in the logs.
>
>
> Diffs
> -----
>
> prism/src/main/java/org/apache/falcon/Main.java 96e003c
> rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunConsumer.java 9ee94c5
> rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunHandler.java f019737
> rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java c2cb09e
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java c6bc36f
> rerun/src/main/java/org/apache/falcon/rerun/queue/ActiveMQueue.java 021e4cc
> rerun/src/main/java/org/apache/falcon/rerun/service/LateRunService.java 2bb198b
>
> Diff: https://reviews.apache.org/r/38166/diff/
>
>
> Testing
> -------
>
> I have started and stopped the server. Logs look clean and each service gets shutdown properly.
> Handled activemq brokerservice and server stops separately.
>
> Below is how the logs are looking after the patch.
>
> 2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ calling shutdown hook (Main:68)
> 2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ Stopped SocketConnector@0.0.0.0:15000 (log:67)
> 2015-09-07 18:08:02,922 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:65)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:65)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Closing topicSubscriber on topic : FALCON.ENTITY.TOPIC (JMSMessageConsumer:149)
> 2015-09-07 18:08:02,943 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:65)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:58)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:58)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ RetryHandler thread destroyed (RetryService:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:58)
> 2015-09-07 18:08:02,963 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
> 2015-09-07 18:08:02,964 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
> 2015-09-07 18:08:02,965 INFO - [LaterunHandler:] ~ Rerun handler daemon has been interrupted (AbstractRerunConsumer:62)
> 2015-09-07 18:08:02,966 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
> 2015-09-07 18:08:02,967 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,978 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
> 2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
> 2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ LateRun thread destroyed (LateRunService:70)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:65)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:58)
> 2015-09-07 18:08:02,985 INFO - [Thread-0:] ~ Shutting down graph db (MetadataMappingService:202)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:65)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.LogCleanupService (ServiceInitializer:58)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Falcon log cleanup service destroyed (LogCleanupService:80)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.LogCleanupService (ServiceInitializer:65)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~
> ############################################
> Falcon Server (SHUTDOWN)
> ############################################ (ContextStartupListener:94)
> 2015-09-07 18:08:03,047 INFO - [Thread-0:] ~ ActiveMQ Message Broker (localhost, ID:groot-54663-1441649273016-0:1) is shutting down (BrokerService:560)
> 2015-09-07 18:08:03,048 INFO - [Thread-0:] ~ Connector vm://localhost Stopped (TransportConnector:288)
> 2015-09-07 18:08:03,632 INFO - [Thread-0:] ~ Connector tcp://groot:61616 Stopped (TransportConnector:288)
> 2015-09-07 18:08:03,641 INFO - [Thread-0:] ~ JobSchedulerStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/scheduler stopped (JobSchedulerStore:286)
> 2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ PListStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/tmp_storage stopped (PListStore:307)
> 2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ Stopping async queue tasks (KahaDBStore:206)
> 2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopping async topic tasks (KahaDBStore:220)
> 2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopped KahaDB (KahaDBStore:246)
> 2015-09-07 18:08:04,048 INFO - [Thread-0:] ~ ActiveMQ JMS Message Broker (localhost, ID:groot-54663-1441649273016-0:1) stopped (BrokerService:628)
>
>
> Thanks,
>
> sandeep samudrala
>
>
Re: Review Request 38166: Add shutdown hook for Falcon
Posted by sandeep samudrala <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38166/
-----------------------------------------------------------
(Updated Sept. 9, 2015, 6:23 a.m.)
Review request for Falcon.
Bugs: FALCON-348
https://issues.apache.org/jira/browse/FALCON-348
Repository: falcon-git
Description
-------
A shutdown hook is missing and services are shutdown in a brute force mode. I'd like to see activemq, titan and such are shutdown gracefully so we don't lose data and see ugly exceptions in the logs.
Diffs (updated)
-----
prism/src/main/java/org/apache/falcon/Main.java 96e003c
rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunConsumer.java 9ee94c5
rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunHandler.java f019737
rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java c2cb09e
rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java c6bc36f
rerun/src/main/java/org/apache/falcon/rerun/queue/ActiveMQueue.java 021e4cc
rerun/src/main/java/org/apache/falcon/rerun/service/LateRunService.java 2bb198b
Diff: https://reviews.apache.org/r/38166/diff/
Testing
-------
I have started and stopped the server. Logs look clean and each service gets shutdown properly.
Handled activemq brokerservice and server stops separately.
Below is how the logs are looking after the patch.
2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ calling shutdown hook (Main:68)
2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ Stopped SocketConnector@0.0.0.0:15000 (log:67)
2015-09-07 18:08:02,922 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:58)
2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:65)
2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:58)
2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:65)
2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:58)
2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Closing topicSubscriber on topic : FALCON.ENTITY.TOPIC (JMSMessageConsumer:149)
2015-09-07 18:08:02,943 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:65)
2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:58)
2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:65)
2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:58)
2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ RetryHandler thread destroyed (RetryService:65)
2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:65)
2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:58)
2015-09-07 18:08:02,963 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
2015-09-07 18:08:02,964 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
2015-09-07 18:08:02,965 INFO - [LaterunHandler:] ~ Rerun handler daemon has been interrupted (AbstractRerunConsumer:62)
2015-09-07 18:08:02,966 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
2015-09-07 18:08:02,967 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
2015-09-07 18:08:02,978 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ LateRun thread destroyed (LateRunService:70)
2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:65)
2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:58)
2015-09-07 18:08:02,985 INFO - [Thread-0:] ~ Shutting down graph db (MetadataMappingService:202)
2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:65)
2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.LogCleanupService (ServiceInitializer:58)
2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Falcon log cleanup service destroyed (LogCleanupService:80)
2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.LogCleanupService (ServiceInitializer:65)
2015-09-07 18:08:03,046 INFO - [Thread-0:] ~
############################################
Falcon Server (SHUTDOWN)
############################################ (ContextStartupListener:94)
2015-09-07 18:08:03,047 INFO - [Thread-0:] ~ ActiveMQ Message Broker (localhost, ID:groot-54663-1441649273016-0:1) is shutting down (BrokerService:560)
2015-09-07 18:08:03,048 INFO - [Thread-0:] ~ Connector vm://localhost Stopped (TransportConnector:288)
2015-09-07 18:08:03,632 INFO - [Thread-0:] ~ Connector tcp://groot:61616 Stopped (TransportConnector:288)
2015-09-07 18:08:03,641 INFO - [Thread-0:] ~ JobSchedulerStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/scheduler stopped (JobSchedulerStore:286)
2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ PListStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/tmp_storage stopped (PListStore:307)
2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ Stopping async queue tasks (KahaDBStore:206)
2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopping async topic tasks (KahaDBStore:220)
2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopped KahaDB (KahaDBStore:246)
2015-09-07 18:08:04,048 INFO - [Thread-0:] ~ ActiveMQ JMS Message Broker (localhost, ID:groot-54663-1441649273016-0:1) stopped (BrokerService:628)
Thanks,
sandeep samudrala
Re: Review Request 38166: Add shutdown hook for Falcon
Posted by pavan kumar kolamuri <pa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38166/#review97995
-----------------------------------------------------------
rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java (line 98)
<https://reviews.apache.org/r/38166/#comment154136>
Order should be interrupt and close right ?
- pavan kumar kolamuri
On Sept. 8, 2015, 4:38 a.m., sandeep samudrala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38166/
> -----------------------------------------------------------
>
> (Updated Sept. 8, 2015, 4:38 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-348
> https://issues.apache.org/jira/browse/FALCON-348
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> A shutdown hook is missing and services are shutdown in a brute force mode. I'd like to see activemq, titan and such are shutdown gracefully so we don't lose data and see ugly exceptions in the logs.
>
>
> Diffs
> -----
>
> prism/src/main/java/org/apache/falcon/Main.java 96e003c
> rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunConsumer.java 9ee94c5
> rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunHandler.java f019737
> rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java c2cb09e
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java c6bc36f
> rerun/src/main/java/org/apache/falcon/rerun/queue/ActiveMQueue.java 021e4cc
> rerun/src/main/java/org/apache/falcon/rerun/service/LateRunService.java 2bb198b
>
> Diff: https://reviews.apache.org/r/38166/diff/
>
>
> Testing
> -------
>
> I have started and stopped the server. Logs look clean and each service gets shutdown properly.
> Handled activemq brokerservice and server stops separately.
>
> Below is how the logs are looking after the patch.
>
> 2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ calling shutdown hook (Main:68)
> 2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ Stopped SocketConnector@0.0.0.0:15000 (log:67)
> 2015-09-07 18:08:02,922 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:65)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:65)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:58)
> 2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Closing topicSubscriber on topic : FALCON.ENTITY.TOPIC (JMSMessageConsumer:149)
> 2015-09-07 18:08:02,943 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:65)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:58)
> 2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:58)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ RetryHandler thread destroyed (RetryService:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:65)
> 2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:58)
> 2015-09-07 18:08:02,963 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
> 2015-09-07 18:08:02,964 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
> 2015-09-07 18:08:02,965 INFO - [LaterunHandler:] ~ Rerun handler daemon has been interrupted (AbstractRerunConsumer:62)
> 2015-09-07 18:08:02,966 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
> 2015-09-07 18:08:02,967 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,978 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
> 2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
> 2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ LateRun thread destroyed (LateRunService:70)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:65)
> 2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:58)
> 2015-09-07 18:08:02,985 INFO - [Thread-0:] ~ Shutting down graph db (MetadataMappingService:202)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:65)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.LogCleanupService (ServiceInitializer:58)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Falcon log cleanup service destroyed (LogCleanupService:80)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.LogCleanupService (ServiceInitializer:65)
> 2015-09-07 18:08:03,046 INFO - [Thread-0:] ~
> ############################################
> Falcon Server (SHUTDOWN)
> ############################################ (ContextStartupListener:94)
> 2015-09-07 18:08:03,047 INFO - [Thread-0:] ~ ActiveMQ Message Broker (localhost, ID:groot-54663-1441649273016-0:1) is shutting down (BrokerService:560)
> 2015-09-07 18:08:03,048 INFO - [Thread-0:] ~ Connector vm://localhost Stopped (TransportConnector:288)
> 2015-09-07 18:08:03,632 INFO - [Thread-0:] ~ Connector tcp://groot:61616 Stopped (TransportConnector:288)
> 2015-09-07 18:08:03,641 INFO - [Thread-0:] ~ JobSchedulerStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/scheduler stopped (JobSchedulerStore:286)
> 2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ PListStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/tmp_storage stopped (PListStore:307)
> 2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ Stopping async queue tasks (KahaDBStore:206)
> 2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopping async topic tasks (KahaDBStore:220)
> 2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopped KahaDB (KahaDBStore:246)
> 2015-09-07 18:08:04,048 INFO - [Thread-0:] ~ ActiveMQ JMS Message Broker (localhost, ID:groot-54663-1441649273016-0:1) stopped (BrokerService:628)
>
>
> Thanks,
>
> sandeep samudrala
>
>
Re: Review Request 38166: Add shutdown hook for Falcon
Posted by sandeep samudrala <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38166/
-----------------------------------------------------------
(Updated Sept. 8, 2015, 4:38 a.m.)
Review request for Falcon.
Bugs: FALCON-348
https://issues.apache.org/jira/browse/FALCON-348
Repository: falcon-git
Description
-------
A shutdown hook is missing and services are shutdown in a brute force mode. I'd like to see activemq, titan and such are shutdown gracefully so we don't lose data and see ugly exceptions in the logs.
Diffs
-----
prism/src/main/java/org/apache/falcon/Main.java 96e003c
rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunConsumer.java 9ee94c5
rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunHandler.java f019737
rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java c2cb09e
rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java c6bc36f
rerun/src/main/java/org/apache/falcon/rerun/queue/ActiveMQueue.java 021e4cc
rerun/src/main/java/org/apache/falcon/rerun/service/LateRunService.java 2bb198b
Diff: https://reviews.apache.org/r/38166/diff/
Testing (updated)
-------
I have started and stopped the server. Logs look clean and each service gets shutdown properly.
Handled activemq brokerservice and server stops separately.
Below is how the logs are looking after the patch.
2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ calling shutdown hook (Main:68)
2015-09-07 18:08:02,913 INFO - [Thread-0:] ~ Stopped SocketConnector@0.0.0.0:15000 (log:67)
2015-09-07 18:08:02,922 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:58)
2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.security.AuthenticationInitializationService (ServiceInitializer:65)
2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:58)
2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.workflow.WorkflowJobEndNotificationService (ServiceInitializer:65)
2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:58)
2015-09-07 18:08:02,923 INFO - [Thread-0:] ~ Closing topicSubscriber on topic : FALCON.ENTITY.TOPIC (JMSMessageConsumer:149)
2015-09-07 18:08:02,943 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.ProcessSubscriberService (ServiceInitializer:65)
2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:58)
2015-09-07 18:08:02,959 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:65)
2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:58)
2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ RetryHandler thread destroyed (RetryService:65)
2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.RetryService (ServiceInitializer:65)
2015-09-07 18:08:02,960 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:58)
2015-09-07 18:08:02,963 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
2015-09-07 18:08:02,964 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
2015-09-07 18:08:02,965 INFO - [LaterunHandler:] ~ Rerun handler daemon has been interrupted (AbstractRerunConsumer:62)
2015-09-07 18:08:02,966 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
2015-09-07 18:08:02,967 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
2015-09-07 18:08:02,978 INFO - [Thread-0:] ~ Closing queue for broker=tcp://localhost:61616, destinationfalcon.late.queue (DelayedQueue:132)
2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close producer (MessagingUtil:76)
2015-09-07 18:08:02,980 INFO - [Thread-0:] ~ Attempting to close consumer (MessagingUtil:87)
2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Attempting to close connection (MessagingUtil:65)
2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ LateRun thread destroyed (LateRunService:70)
2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.rerun.service.LateRunService (ServiceInitializer:65)
2015-09-07 18:08:02,984 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:58)
2015-09-07 18:08:02,985 INFO - [Thread-0:] ~ Shutting down graph db (MetadataMappingService:202)
2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:65)
2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Destroying service: org.apache.falcon.service.LogCleanupService (ServiceInitializer:58)
2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Falcon log cleanup service destroyed (LogCleanupService:80)
2015-09-07 18:08:03,046 INFO - [Thread-0:] ~ Service destroyed: org.apache.falcon.service.LogCleanupService (ServiceInitializer:65)
2015-09-07 18:08:03,046 INFO - [Thread-0:] ~
############################################
Falcon Server (SHUTDOWN)
############################################ (ContextStartupListener:94)
2015-09-07 18:08:03,047 INFO - [Thread-0:] ~ ActiveMQ Message Broker (localhost, ID:groot-54663-1441649273016-0:1) is shutting down (BrokerService:560)
2015-09-07 18:08:03,048 INFO - [Thread-0:] ~ Connector vm://localhost Stopped (TransportConnector:288)
2015-09-07 18:08:03,632 INFO - [Thread-0:] ~ Connector tcp://groot:61616 Stopped (TransportConnector:288)
2015-09-07 18:08:03,641 INFO - [Thread-0:] ~ JobSchedulerStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/scheduler stopped (JobSchedulerStore:286)
2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ PListStore:/home/dataqa/sandeep/falcon-0.8-SNAPSHOT/data/localhost/tmp_storage stopped (PListStore:307)
2015-09-07 18:08:03,642 INFO - [Thread-0:] ~ Stopping async queue tasks (KahaDBStore:206)
2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopping async topic tasks (KahaDBStore:220)
2015-09-07 18:08:03,643 INFO - [Thread-0:] ~ Stopped KahaDB (KahaDBStore:246)
2015-09-07 18:08:04,048 INFO - [Thread-0:] ~ ActiveMQ JMS Message Broker (localhost, ID:groot-54663-1441649273016-0:1) stopped (BrokerService:628)
Thanks,
sandeep samudrala