You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by giannis_k <ik...@yahoo.gr> on 2017/05/11 12:44:14 UTC
Camel spring boot destroys beans with wrong order.
Hello,
I am using camel 2.18.2 with camel spring boot starter and spring versions:
<spring.boot-version>1.4.1.RELEASE</spring.boot-version>
<spring.version>4.3.3.RELEASE</spring.version>
We consume messages from a kafka component and we shutdown the application,
some beans are destroyed before the CamelContext causing messages to be lost
because either e seda component does not exist or the connection to the
database has been closed. Logs from an example:
15:40:48.966 [Thread-3] INFO o.s.c.a.AnnotationConfigApplicationContext -
Closing
org.springframework.context.annotation.AnnotationConfigApplicationContext@4218d6a3:
startup date [Wed May 10 15:40:30 EEST 2017]; root of context hierarchy
15:40:48.966 [Thread-4] INFO o.a.c.m.MainSupport$HangupInterceptor -
Received hang up - stopping the main instance.
15:40:48.966 [Thread-4] INFO o.a.camel.spring.SpringCamelContext - Apache
Camel 2.18.2 (CamelContext: camel) is shutting down
15:40:48.967 [Thread-3] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
Returning cached instance of singleton bean 'sqlSessionFactory'
15:40:48.967 [Thread-3] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
Returning cached instance of singleton bean 'sqlSessionFactorySecondary'
....
15:40:48.968 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
destroy method 'shutdown' on bean with name 'direct-component'
15:40:48.968 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
destroy method 'shutdown' on bean with name 'seda-component'
15:40:48.973 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
destroy() on bean with name 'mbeanExporter'
15:40:48.973 [Thread-3] INFO o.s.j.e.a.AnnotationMBeanExporter -
Unregistering JMX-exposed beans on shutdown
15:40:48.973 [Thread-4] INFO o.a.c.impl.DefaultShutdownStrategy - Starting
to graceful shutdown 21 routes (timeout 300 seconds)
15:40:48.974 [Thread-3] INFO o.s.j.e.a.AnnotationMBeanExporter -
Unregistering JMX-exposed beans
15:40:48.980 [Thread-3] DEBUG o.s.c.a.CommonAnnotationBeanPostProcessor -
Invoking destroy method on bean 'applicationController': private void
org.apache.camel.spring.boot.CamelSpringBootApplicationController.destroy()
15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
destroy() on bean with name 'consumerTemplate'
15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
destroy() on bean with name 'fluentTemplate'
15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
destroy() on bean with name 'template'
15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
Retrieved dependent beans for bean 'camel': [typeConverter,
springTypeConverter, properties,
org.apache.camel.component.kafka.KafkaComponent, constant-language,
org.apache.camel.component.file.FileComponent,
org.apache.camel.component.quartz2.QuartzComponent,
org.apache.camel.component.zookeeper.ZooKeeperComponent, header-language]
15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
destroy() on bean with name 'camel'
15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
Retrieved dependent beans for bean 'ehcache': [cacheManager]
15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
Retrieved dependent beans for bean 'cacheManager':
[org.springframework.cache.interceptor.CacheInterceptor#0]
15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
destroy() on bean with name 'ehcache'
15:40:48.980 [Thread-3] INFO o.s.c.e.EhCacheManagerFactoryBean - Shutting
down EhCache CacheManager
...
15:40:49.017 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
destroy method 'close' on bean with name 'dataSourceSecondary'
15:40:49.017 [Thread-3] INFO com.zaxxer.hikari.pool.HikariPool -
HikariPool-2 - Close initiated...
org.apache.camel.component.seda.SedaConsumerNotAvailableException: No queue
available on endpoint: seda://placed.
at
org.apache.camel.component.seda.SedaProducer.addToQueue(SedaProducer.java:223)
at
org.apache.camel.component.seda.SedaProducer.process(SedaProducer.java:169)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
I can see a similar case in here
<http://camel.465427.n5.nabble.com/OnApplicationEvent-not-called-with-CamelAutoConfiguration-td5798802.html>
however the workaround does not work for me. Is there a possible workaround
for this issue?
Thank you,
Giannis
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-spring-boot-destroys-beans-with-wrong-order-tp5799304.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel spring boot destroys beans with wrong order.
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Cameleers, Giannis,
we have 2.20 snapshots built at the snapshot repostory[1], other than
a incompatibility with `noStart` flag -- that I don't think is much
used outside of tests, I believe it should fix this.
We are considering to backport this to next patch release of 2.19 (2.19.1).
Any feedback is welcome :)
thanks,
zoran
[1] https://repository.apache.org/content/repositories/snapshots/org/apache/camel/
On Fri, May 12, 2017 at 5:04 PM, Zoran Regvart <zo...@regvart.com> wrote:
> Hi Giannis,
> thank you for testing and providing logs, this is now something we can
> turn into tests in order to find the best solution,
>
> thanks :)
>
> zoran
>
> On Fri, May 12, 2017 at 4:26 PM, giannis_k <ik...@yahoo.gr> wrote:
>> I didn't use SmartLifeCycle for a particular reason, it was just a leftover
>> of trying stuff :).
>> As for the fix in the PR I tried it and it didn't work, I see the same
>> behaviour as with the workaround without the loop.
>
> --
> Zoran Regvart
--
Zoran Regvart
Re: Camel spring boot destroys beans with wrong order.
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Giannis,
thank you for testing and providing logs, this is now something we can
turn into tests in order to find the best solution,
thanks :)
zoran
On Fri, May 12, 2017 at 4:26 PM, giannis_k <ik...@yahoo.gr> wrote:
> I didn't use SmartLifeCycle for a particular reason, it was just a leftover
> of trying stuff :).
> As for the fix in the PR I tried it and it didn't work, I see the same
> behaviour as with the workaround without the loop.
--
Zoran Regvart
Re: Camel spring boot destroys beans with wrong order.
Posted by giannis_k <ik...@yahoo.gr>.
Hey Zoran,
I didn't use SmartLifeCycle for a particular reason, it was just a leftover
of trying stuff :).
As for the fix in the PR I tried it and it didn't work, I see the same
behaviour as with the workaround without the loop. Here are some logs:
16:58:32.026 [Thread-3] INFO o.a.c.m.MainSupport$HangupInterceptor -
Received hang up - stopping the main instance.
16:58:32.026 [Thread-3] DEBUG org.apache.camel.main.MainSupport - Controller
is shutting down CamelContext
16:58:32.026 [Thread-4] INFO o.s.c.a.AnnotationConfigApplicationContext -
Closing
org.springframework.context.annotation.AnnotationConfigApplicationContext@51399530:
startup date [Fri May 12 16:58:12 EEST 2017]; root of context hierarchy
16:58:32.026 [CamelMainRunController] DEBUG
o.a.c.s.b.CamelSpringBootApplicationController - Waiting for CamelContext to
complete shutdown
16:58:32.026 [Thread-3] INFO o.a.camel.spring.SpringCamelContext - Apache
Camel 2.20.0-SNAPSHOT (CamelContext: camel) is shutting down
16:58:32.028 [Thread-4] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
Returning cached instance of singleton bean 'sqlSessionFactory'
16:58:32.029 [Thread-4] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
Returning cached instance of singleton bean 'sqlSessionFactorySecondary'
16:58:32.029 [Thread-4] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
Returning cached instance of singleton bean 'camel'
16:58:32.031 [Thread-3] INFO o.a.c.impl.DefaultShutdownStrategy - Starting
to graceful shutdown 10 routes (timeout 300 seconds)
....
16:58:32.033 [Thread-4] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
destroy method 'shutdown' on bean with name 'direct-component'
16:58:32.040 [Thread-4] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
destroy method 'shutdown' on bean with name 'file-component'
...
16:58:32.067 [Thread-4] INFO com.zaxxer.hikari.pool.HikariPool - HikariCP
pool HikariPool-1 is shutting down.
16:58:32.107 [Thread #2 - ShutdownTask] INFO org.apache.zookeeper.ZooKeeper
- Session: 0x15ac81624a033cd closed
....
16:58:33.080 [Thread-3] INFO o.a.camel.spring.SpringCamelContext - Apache
Camel 2.20.0-SNAPSHOT (CamelContext: camel) uptime 8.856 seconds
16:58:33.080 [Thread-3] INFO o.a.camel.spring.SpringCamelContext - Apache
Camel 2.20.0-SNAPSHOT (CamelContext: camel) is shutdown in 1.054 seconds
16:58:33.080 [CamelMainRunController] DEBUG
o.a.c.s.b.CamelSpringBootApplicationController - CamelContext shutdown
complete.
As you can see the beans are destroyed before camel context is shut down. I
said that the shutdown of routes is done asychronously because I noticed in
the logs that a different thread is used for shutting down routes (Thread
#2), a different thread by DefaultShutdownStrategy (Thread-3) and a
different thread for destroying Spring beans (Thread-4).
Giannis
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-spring-boot-destroys-beans-with-wrong-order-tp5799304p5799390.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel spring boot destroys beans with wrong order.
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Giannis,
I don't think you need that while loop, the CamelContext::stop method
should block until CamelContext is stopped, and is there a reason you
implemented SmartLifecycle instaed of Lifeycle? I ask because we had
that discussion on the PR and came to conclusion that SmartLifecycle
in our use-case does not provide any advantage over Lifecycle.
In order to test the code as it is in the PR, you would need to get
the source code from the GitHub repository, and build that branch's
version of Camel, something like:
$ git clone https://github.com/apache/camel.git
$ git fetch origin pull/1685/head:pr1685
$ git checkout pr1685
$ mvn -Pfastinstall install
The last command should build (after ~20 minutes or so) the jar files
and place them in your local Maven repository so that you can
reference those with the version 2.20.0-SNAPSHOT.
zoran
On Fri, May 12, 2017 at 11:34 AM, giannis_k <ik...@yahoo.gr> wrote:
> Hello Zoran,
>
> thank you for the input it was really helpful. I tried it and although I can
> see that now the camel context definately shuts down first, the other beans
> are still shut down before camel context is closed probably because the
> shutdown of routes happen asychronously. What I ended up doing to overcome
> this issue is to place a loop inside my 'SmartLifecycle' bean to wait for
> the camel context to close. Here is my workaround:
>
> public class SpringBootCamelFix implements SmartLifecycle {
>
> private static final Logger log =
> LoggerFactory.getLogger(SpringBootCamelFix.class);
>
> @Autowired
> DefaultCamelContext context;
>
>
> @Override
> public void start() {}
>
> @Override
> public void stop() {
>
> if (!isRunning()) {
> log.info("Camel context already stopped");
> return;
> }
>
> log.info("Stopping camel context. Will wait until it is actually
> stopped");
>
> try {
> context.stop();
> } catch (Exception e) {
> e.printStackTrace();
> }
>
> while(context.isStarted());
> }
>
> @Override
> public boolean isRunning() {
> return context.isStarted();
> }
>
> @Override
> public int getPhase() {
> return Integer.MAX_VALUE;
> }
>
> @Override
> public boolean isAutoStartup() {
> return false;
> }
>
> @Override
> public void stop(Runnable callback) {
> stop();
> callback.run();
> }
> }
>
>
> I would also like to test your changes however where can I get a jar with
> your changes in order to test it locally?
>
> Thanks,
> Giannis
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-spring-boot-destroys-beans-with-wrong-order-tp5799304p5799359.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Zoran Regvart
Re: Camel spring boot destroys beans with wrong order.
Posted by giannis_k <ik...@yahoo.gr>.
Hello Zoran,
thank you for the input it was really helpful. I tried it and although I can
see that now the camel context definately shuts down first, the other beans
are still shut down before camel context is closed probably because the
shutdown of routes happen asychronously. What I ended up doing to overcome
this issue is to place a loop inside my 'SmartLifecycle' bean to wait for
the camel context to close. Here is my workaround:
public class SpringBootCamelFix implements SmartLifecycle {
private static final Logger log =
LoggerFactory.getLogger(SpringBootCamelFix.class);
@Autowired
DefaultCamelContext context;
@Override
public void start() {}
@Override
public void stop() {
if (!isRunning()) {
log.info("Camel context already stopped");
return;
}
log.info("Stopping camel context. Will wait until it is actually
stopped");
try {
context.stop();
} catch (Exception e) {
e.printStackTrace();
}
while(context.isStarted());
}
@Override
public boolean isRunning() {
return context.isStarted();
}
@Override
public int getPhase() {
return Integer.MAX_VALUE;
}
@Override
public boolean isAutoStartup() {
return false;
}
@Override
public void stop(Runnable callback) {
stop();
callback.run();
}
}
I would also like to test your changes however where can I get a jar with
your changes in order to test it locally?
Thanks,
Giannis
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-spring-boot-destroys-beans-with-wrong-order-tp5799304p5799359.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel spring boot destroys beans with wrong order.
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Giannis,
we have a pull request[1] in review that should help with such issues,
would you mind taking a look and commenting on it or trying out and
confirming that it helps in your case?
As a workaround you could create your own Spring lifecycle bean, it
should implement `Lifecycle` and `Phased` as does `SpringCamelContext`
in that pull request and in `stop` method perform
CamelContext::shutdown.
zoran
[1] https://github.com/apache/camel/pull/1685
On Thu, May 11, 2017 at 2:44 PM, giannis_k <ik...@yahoo.gr> wrote:
> Hello,
>
> I am using camel 2.18.2 with camel spring boot starter and spring versions:
>
> <spring.boot-version>1.4.1.RELEASE</spring.boot-version>
> <spring.version>4.3.3.RELEASE</spring.version>
>
> We consume messages from a kafka component and we shutdown the application,
> some beans are destroyed before the CamelContext causing messages to be lost
> because either e seda component does not exist or the connection to the
> database has been closed. Logs from an example:
>
>
> 15:40:48.966 [Thread-3] INFO o.s.c.a.AnnotationConfigApplicationContext -
> Closing
> org.springframework.context.annotation.AnnotationConfigApplicationContext@4218d6a3:
> startup date [Wed May 10 15:40:30 EEST 2017]; root of context hierarchy
> 15:40:48.966 [Thread-4] INFO o.a.c.m.MainSupport$HangupInterceptor -
> Received hang up - stopping the main instance.
> 15:40:48.966 [Thread-4] INFO o.a.camel.spring.SpringCamelContext - Apache
> Camel 2.18.2 (CamelContext: camel) is shutting down
> 15:40:48.967 [Thread-3] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
> Returning cached instance of singleton bean 'sqlSessionFactory'
> 15:40:48.967 [Thread-3] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
> Returning cached instance of singleton bean 'sqlSessionFactorySecondary'
> ....
> 15:40:48.968 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
> destroy method 'shutdown' on bean with name 'direct-component'
> 15:40:48.968 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
> destroy method 'shutdown' on bean with name 'seda-component'
> 15:40:48.973 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
> destroy() on bean with name 'mbeanExporter'
> 15:40:48.973 [Thread-3] INFO o.s.j.e.a.AnnotationMBeanExporter -
> Unregistering JMX-exposed beans on shutdown
> 15:40:48.973 [Thread-4] INFO o.a.c.impl.DefaultShutdownStrategy - Starting
> to graceful shutdown 21 routes (timeout 300 seconds)
> 15:40:48.974 [Thread-3] INFO o.s.j.e.a.AnnotationMBeanExporter -
> Unregistering JMX-exposed beans
> 15:40:48.980 [Thread-3] DEBUG o.s.c.a.CommonAnnotationBeanPostProcessor -
> Invoking destroy method on bean 'applicationController': private void
> org.apache.camel.spring.boot.CamelSpringBootApplicationController.destroy()
> 15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
> destroy() on bean with name 'consumerTemplate'
> 15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
> destroy() on bean with name 'fluentTemplate'
> 15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
> destroy() on bean with name 'template'
> 15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
> Retrieved dependent beans for bean 'camel': [typeConverter,
> springTypeConverter, properties,
> org.apache.camel.component.kafka.KafkaComponent, constant-language,
> org.apache.camel.component.file.FileComponent,
> org.apache.camel.component.quartz2.QuartzComponent,
> org.apache.camel.component.zookeeper.ZooKeeperComponent, header-language]
> 15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
> destroy() on bean with name 'camel'
> 15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
> Retrieved dependent beans for bean 'ehcache': [cacheManager]
> 15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DefaultListableBeanFactory -
> Retrieved dependent beans for bean 'cacheManager':
> [org.springframework.cache.interceptor.CacheInterceptor#0]
> 15:40:48.980 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
> destroy() on bean with name 'ehcache'
> 15:40:48.980 [Thread-3] INFO o.s.c.e.EhCacheManagerFactoryBean - Shutting
> down EhCache CacheManager
> ...
> 15:40:49.017 [Thread-3] DEBUG o.s.b.f.s.DisposableBeanAdapter - Invoking
> destroy method 'close' on bean with name 'dataSourceSecondary'
> 15:40:49.017 [Thread-3] INFO com.zaxxer.hikari.pool.HikariPool -
> HikariPool-2 - Close initiated...
>
> org.apache.camel.component.seda.SedaConsumerNotAvailableException: No queue
> available on endpoint: seda://placed.
> at
> org.apache.camel.component.seda.SedaProducer.addToQueue(SedaProducer.java:223)
> at
> org.apache.camel.component.seda.SedaProducer.process(SedaProducer.java:169)
> at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
>
>
> I can see a similar case in here
> <http://camel.465427.n5.nabble.com/OnApplicationEvent-not-called-with-CamelAutoConfiguration-td5798802.html>
> however the workaround does not work for me. Is there a possible workaround
> for this issue?
>
> Thank you,
> Giannis
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-spring-boot-destroys-beans-with-wrong-order-tp5799304.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Zoran Regvart