You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by codan84 <da...@live.com> on 2015/11/12 15:35:02 UTC
Spring-Boot + Camel + producerTemplate ssh spawning thousands of
threads
I did write a simple app using Spring Boot (1.2.7.RELEASE) and Apache Camel
(2.15.0). The app is simple and has only 1 route: a timer will invoke a
method on a bean every 1s. The method invoked will use ProducerTemplate to
ssh into a remote machine, execute a small script, and print out the output
to the console. Simple, right?
However, when profiling this, I can see the number of threads! It seems like
any threads created for the ssh are never killed, but parked instead.
Because of that I run OOM pretty quickly with an error:
Exception in thread "Thread-341" java.lang.OutOfMemoryError: unable to
create new native thread
More description and profiler results can be found on stackoverflow:
http://stackoverflow.com/questions/33671567/spring-boot-camel-producertemplate-memory-leak
I think this is a bug, as from the documentation I can't see that I am using
the ProducerTemplate wrongly.
Or am I? If so, please tell me. If I am doing things right and it seems like
a bug, I will submit it...
--
View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands
of threads
Posted by codan84 <da...@live.com>.
I have tried similar thing, but with sftp rather than ssh. Everything else is
exactly the same:
producerTemplate.sendBodyAndHeader(
"sftp://_target_machine_url_?username=_username_" +
"&privateKeyFile=" + sshKeyPath +
"&knownHostsFile=" + knownHosts +
"&preferredAuthentications=publickey",
new ByteArrayInputStream(new String("Random
junk").getBytes()),
"CamelFileName", "/home/_username_/test_file.txt"
);
And this works fine. So it is not the way I am using ProducerTemplate. It is
a bug with ssh when using producer template, or "requestBody" method...
--
View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773751.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands
of threads
Posted by codan84 <da...@live.com>.
In that case starting/stopping it should in theory release threads, right?
I tried start->requestBody->stop, without effect, still all threads keep
running. I also tried to instantiate new template for each call, start it
and stop at the end, still the same.
--
View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773750.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads
Posted by Claus Ibsen <cl...@gmail.com>.
Maybe your bean with the producer template is not singleton scoped, so
spring creates a new instance per message.
On Thu, Nov 12, 2015 at 4:10 PM, codan84 <da...@live.com> wrote:
> Hmm Interesting, this indeed fixes the problem. I am forced to use a
> producerTemplate tho... Any ideas what am I doing wrong?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773747.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of
threads
Posted by Jakub Korab <ja...@gmail.com>.
This bit is definitely a bug. Whether it's something in the component,
or the SshClient library needs to be investigated. You should log this
one in a Jira.
On 12/11/15 16:42, codan84 wrote:
> Indeed the ClientSession of SshClient is being started, but never finished.
> Here are logs:
>
> 2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-1] INFO
> o.a.s.c.session.ClientSessionImpl - Client session created
> 2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-2] INFO
> o.a.s.c.session.ClientSessionImpl - Server version string:
> SSH-1.99-OpenSSH_3.9p1
> 2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-3] INFO
> o.a.s.c.session.ClientSessionImpl - Kex: server->client aes128-ctr hmac-sha1
> none
> 2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-3] INFO
> o.a.s.c.session.ClientSessionImpl - Kex: client->server aes128-ctr hmac-sha1
> none
> 2015-11-12 16:39:09 [sshd-SshClient[68c05218]-nio2-thread-5] WARN
> o.a.s.c.k.AcceptAllServerKeyVerifier - Server at *** presented unverified
> DSA key: ***
> ----
> Script output :)
> ----
>
> This just repeats for every call.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773762.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands
of threads
Posted by codan84 <da...@live.com>.
Indeed the ClientSession of SshClient is being started, but never finished.
Here are logs:
2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-1] INFO
o.a.s.c.session.ClientSessionImpl - Client session created
2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-2] INFO
o.a.s.c.session.ClientSessionImpl - Server version string:
SSH-1.99-OpenSSH_3.9p1
2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-3] INFO
o.a.s.c.session.ClientSessionImpl - Kex: server->client aes128-ctr hmac-sha1
none
2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-3] INFO
o.a.s.c.session.ClientSessionImpl - Kex: client->server aes128-ctr hmac-sha1
none
2015-11-12 16:39:09 [sshd-SshClient[68c05218]-nio2-thread-5] WARN
o.a.s.c.k.AcceptAllServerKeyVerifier - Server at *** presented unverified
DSA key: ***
----
Script output :)
----
This just repeats for every call.
--
View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773762.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of
threads
Posted by Jakub Korab <ja...@gmail.com>.
If the threads are not being cleaned up, then it would seem that there
is a leak in the SshClient library - this is confirmed by the screenshot
of the threads that you attached on SO; they are all associated with
SshClient. Can you verify that the client is in fact shutting down by
set the logging level for org.apache.sshd.common.util to DEBUG?
As to the documentation around the ProducerTemplate and singletons,
absolutely. As they say, contributions are most welcome.
On 12/11/15 16:15, codan84 wrote:
> I tried your suggestion:
>
> ProducerTemplate producerTemplate =
> camelContext.createProducerTemplate();
> producerTemplate.start();
> String response = producerTemplate.requestBody("ssh://....",
> String.class);
> producerTemplate.stop();
>
> There is no difference. Still the number of threads grows.
> I would disagree that this is not a bug. Especially with a Spring-Boot
> application, where the app will usually run on a server forever, this is an
> issue. It is unfortunate how Camel handles non-singleton endpoints, but it
> should either be very well documented (bold, red etc), or a different
> approach to non-singleton endpoints worked out.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773755.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands
of threads
Posted by codan84 <da...@live.com>.
I tried your suggestion:
ProducerTemplate producerTemplate =
camelContext.createProducerTemplate();
producerTemplate.start();
String response = producerTemplate.requestBody("ssh://....",
String.class);
producerTemplate.stop();
There is no difference. Still the number of threads grows.
I would disagree that this is not a bug. Especially with a Spring-Boot
application, where the app will usually run on a server forever, this is an
issue. It is unfortunate how Camel handles non-singleton endpoints, but it
should either be very well documented (bold, red etc), or a different
approach to non-singleton endpoints worked out.
--
View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773755.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of
threads
Posted by Jakub Korab <ja...@gmail.com>.
When you use the SSH producer endpoint in a to(..) statement, one
instance of the endpoint is created within the route, its lifecycle is
tied to the CamelContext and any resources used by it will be cleaned up
at shutdown. The endpoint creates an instance of an SshClient
(underlying library), and closes it when the context shuts down.
When you use it through a ProducerTemplate, each time you access the
send() method it creates a new instance of the SshEndpoint, allocating
memory on the way by creating a new underlying SshClient instance. This
behaviour is dictated by SshEndpoint. isSingleton(), which returns false
as the underlying client library is not thread-safe. You need to close
the ProducerTemplate in order to close the SshClient instance (note,
nothing to do with the SSH connection itself).
As a workaround, you might consider having the CamelContext injected
into your bean, and for each request do:
ProducerTemplate template = context.createProducerTemplate();
template.sendBody(...);
template.close();
It's expensive, but it will do the job.
I don't think this is a bug, if seems to be a side-effect of how a
ProducerTemplate works with non-singleton endpoints. If there were
multiple threads, then the ProducerTemplate would have to create
multiple endpoints for the underlying library to work correctly, and for
the threads to not trip over each other. In your case, only one thread
will ever be using that one ProducerTemplate, so it doesn't make sense.
The ProducerTemplate can't know in advance how many threads will be
accessing it.
On 12/11/15 15:10, codan84 wrote:
> Hmm Interesting, this indeed fixes the problem. I am forced to use a
> producerTemplate tho... Any ideas what am I doing wrong?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773747.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands
of threads
Posted by codan84 <da...@live.com>.
Hmm Interesting, this indeed fixes the problem. I am forced to use a
producerTemplate tho... Any ideas what am I doing wrong?
--
View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773747.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of
threads
Posted by Jakub Korab <ja...@gmail.com>.
To work out if it's your use of the ProducerTemplate, try the following
route instead and see whether the leak persists:
from("timer://foo?period=1000")
.transform().constant("/home/_username_/some_temp_script.sh")
.to("ssh://_remote.machine.url.here_?username=_username_&keyPairProvider=#keyPairProvider")
.log("Received: ${body}");
On 12/11/15 14:35, codan84 wrote:
> I did write a simple app using Spring Boot (1.2.7.RELEASE) and Apache Camel
> (2.15.0). The app is simple and has only 1 route: a timer will invoke a
> method on a bean every 1s. The method invoked will use ProducerTemplate to
> ssh into a remote machine, execute a small script, and print out the output
> to the console. Simple, right?
> However, when profiling this, I can see the number of threads! It seems like
> any threads created for the ssh are never killed, but parked instead.
> Because of that I run OOM pretty quickly with an error:
>
> Exception in thread "Thread-341" java.lang.OutOfMemoryError: unable to
> create new native thread
>
> More description and profiler results can be found on stackoverflow:
> http://stackoverflow.com/questions/33671567/spring-boot-camel-producertemplate-memory-leak
>
> I think this is a bug, as from the documentation I can't see that I am using
> the ProducerTemplate wrongly.
> Or am I? If so, please tell me. If I am doing things right and it seems like
> a bug, I will submit it...
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741.html
> Sent from the Camel - Users mailing list archive at Nabble.com.