You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomee.apache.org by "Niall.Emmerson" <Ni...@visitscotland.com> on 2015/05/20 15:12:07 UTC

TomEE blocks additional EJBD connections after first 10

Greetings All

I'm having a problem with TomEE (1.7.1) when using Tomcat (8) to connect to remote beans via EJBD which I'm beginning to think might be a bug in the OpenEJB code somewhere.

I'm developing and deploying my Tomcat webapp in Eclipse and when I make a change the updated code is automatically deployed to the server and the context reloaded. The first time I start up the server everything works well and I'm able to call the EJBs on TomEE correctly. After making changes to code though, specifically, after the Tomcat servers' context restarts 10 times, on the 11th time TomEE no longer responds to InitialContext lookups and the Tomcat application hangs (it blocks waiting for a response from the TomEE server at org.apache.openejb.client.ProtocolMetadata:101 in the openejb-client jar deployed along with the web app).

I think I've narrowed the problem(s) down to the following.

When the Tomcat context reloads/webapp restarts, the openejb-client jar's connection pool seems to leak into memory so that all the connections opened before the context reloaded remain open (but can't be used). I suspect org.apache.openejb.client.Client remains leaked in memory. When the Tomcat webapp does another lookup, another Client instance is created along with another connection.

When a connection is made to TomEE by Tomcat, the TomEE ServicePool (org.apache.openejb.server) spins up a thread called "OpenEJB.ejbd.X" (where x is an integer). This thread then remains in the runnable state waiting for a socket read input at KeepAliveServer:248. When the client side connection has been leaked into the Tomcat memory this socket read will never return anything and so the thread remainings running. The TomEE ServicePool is making use of a LinkedBlockingQueue. According to the JavaDoc that means that after the Tomcat context has reloaded 10 times (the default ThreadPoolExecutor's threadCore value), the request for a new thread will go into the LinkedBlockingQueue and remain there until the queue is full, at which point an additional thread will be created. This means that after all the core threads are in use, no additional connections can be made to TomEE (unless so many are made as to overwhem the queue).

Something seems wrong with this logic to me?
Surely the TomEE ServicePool shouldn't be using a ThreadPoolExecutor with a queuing strategy that causes subsequent connection attempts to block indefinitely if the threads that it creates also block indefinitely? Or perhaps it's not meant to be the case that the thread it creates block at all?
Also, presumably the client connection pool should terminate the connections if the context is reloaded?

Am I missing something? Some sort of configuration value or something?

Any help/advice would be appreciated.
For Visitor Information<http://www.visitscotland.com>
For information on VisitScotland<http://www.visitscotland.org>
For information on Business Tourism<http://www.conventionscotland.com>
For information on EventScotland<http://www.EventScotland.org>

2015 is Scotland's Year of Food and Drink, celebrating our exceptional natural larder, the role food and drink plays in shaping our economic success and the people, landscapes and culture that make our food heritage so unique.
We invite you to come and experience the Land of Food and Drink. On Twitter? [VisitScotland Twitter]  <https://twitter.com/VisitScotNews> #TasteScotland
Get involved and share your #brilliantmoments [VisitScotland Facebook] <https://www.facebook.com/visitscotland>  [VisitScotland Twitter] <https://twitter.com/visitscotland>  [VisitScotland Instagram] <http://instagram.com/visitscotland>
________________________________
Before you print think about the ENVIRONMENT

This email and any attachments are private and confidential. If you have received this email in error, please immediately notify the sender and permanently remove the email from your system. If you receive this email in error, you must not use, copy or disclose any of the information contained in this e-mail or any attachments.

Any views or opinions expressed in this email and any attachments are those of the sender and do not necessarily represent the views or opinions of VisitScotland. Please note that VisitScotland's email system (including the content of emails and attachments) is subject to monitoring by us for the purposes of security and training.

Tha am post-d seo (agus faidhle sam bith na chois) d?omhair agus sochairichte. Mas ann le mearachd a tha thu air am post-d fhaighinn, cuir fios gun d?il chun neach a chur thugaibh e agus dubh ?s an teachdaireachd-sa sa bhad. Na d?an lethbhreac den fhiosrachadh, na foillsich e ann an d?igh sam bith, na d?an gn?omh ri a linn.

Dh'fhaodadh beachdan pearsanta a bhith gan cur an c?ill sam post-d seo nach eil a' riochdachadh beachdan VisitScotland. Tha siostam post-dealain VisitScotland (an teachdaireachd sam post-dealain seo agus ceangalachain sam bith) ga mheasadh agus ga chl?radh gu tuaireamach airson adhbharan t?arainteachd agus tr?anadh.

Tha VisitScotland a' comhairleachadh gu bheil sibh a' fosgladh puist-d agus cheangalachain aig ur cunnart fh?in.




--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932.html
Sent from the TomEE Users mailing list archive at Nabble.com.

Re: TomEE blocks additional EJBD connections after first 10

Posted by Romain Manni-Bucau <rm...@gmail.com>.
2015-05-22 10:34 GMT+02:00 Niall.Emmerson <Ni...@visitscotland.com>
:

> Romain Manni-Bucau wrote
> >> More than that
> >> though, it would mean that if we ever wanted to have more than 10 Tomcat
> >> servers connecting to the same TomEE server then we'd have the same
> >> problem,
> >>
> >
> > no you would configure the pool to be bigger
>
> Surely that just delays the problem? Let's assume that I need 11 Tomcat's
> to
> act as EJBD clients, I set the default core threads to 15 to give me a bit
> of leeway to add more if I need to. A couple of months down the line,
> something happens but I now need to have 16 EJBD clients. Now I have the
> same problem again but I've got to restart the production instance of TomEE
> in order to get it to pick up a new value for the number of core threads
> and
> as a result everything that relies on that server has to be taken offline
> while the restart occurs. I could configure the number of core threads to
> be
> huge, 200 say, same as the maximum possible threads but then each
> deployment
> of a client application will result in a new thread being created while the
> old thread used by the same client before the new deployment will remain
> parked in the TomEE's memory forever.
>
>
well you need to size the pool according to your need. If you need N thread
for each app and you have M apps then you need N*M threads. The offline
issue is generally solved using a proxy or equivalent solution to be able
to upgrade by column the instances.


>
> Romain Manni-Bucau wrote
> >> the 11th Tomcat wouldn't be able to send requests (though strangely, if
> I
> >> understand this right, if we had 20 Tomcat's connecting to the same
> TomEE
> >> then the 11th would work because of the new thread). While 20 Tomcats
> >> connecting to the same TomEE in production is probably unlikely, 11
> might
> >> not be outwith the bounds of possibility.
> >>
> >> Isn't this something that should be fixed in the OpenEJB code so that
> >> this
> >> sort of situation can't happen? Perhaps creating a new thread if one of
> >> the
> >> existing ones isn't available within a set time period or something?
> >>
> >>
> > I don't really see a bug here since you dont rely on the built in
> > config/setup
>
> Sorry, I don't understand. How am I not using the built in config/setup
> when
> all I've done is change some of the default config values? I actually
> switched on EJBD by using the configuration values you yourself provided in
> another post
> <
> http://tomee-openejb.979440.n4.nabble.com/EJB-socket-deamon-in-tomee-tp4658792p4658793.html
> >
> . I don't understand what situation the current code logic would be useful
> in. Assuming default values, 10 clients are ok, 11-19 aren't and 20-200 are
> but if you do have 20-200 then the last 9 won't work? If the solution is
> just to increase the number of core threads then why have a queue? If I'm
> not meant to use EJBD then why is it part of TomEE and won't the same
> problem exist for standalone OpenEJB but only without the option to get
> around this issue by using HTTP?
>
>
You explicitly enabled the standalone mode so you have the side effects of
the standalone mode which looks good.


> To me it seems that the logic for new connections should be: When a new
> connection is made, create a new thread up until the number of core
> threads,
> after that, if one of the core threads is free then use that for the next
> new connection but if they're all in use, create a new thread and destroy
> whichever thread becomes unused next so that you return to the number of
> core threads when possible.
>
>
that's actually what is done (automatically thanks to the executor
service). The issue is the connection stays pending cause of the use of
blocking IOs (intended in current design). While there is not connection
break we can't know the connection needs to be released.


>
> Anyway, I'll enquire about ways around the firewall issue with HTTP. It
> seems a shame that the EJBD protocol support doesn't work out of the box
> though.
>
>
>
well it is not the ejbd protocol (it works perfectly on HTTP OOTB), it is
the socket transport which is intended to stay connected for performances
reason.

You can still register you own socket connection factory
through org.apache.openejb.client.ConnectionManager#registerFactory,
inheriting from org.apache.openejb.client.SocketConnectionFactory anc
leanup the pool when you undeploy the application.


Small workarounds can be to configure the timeouts:
openejb.client.connection.socket.timeout,
openejb.client.connection.pool.timeout

but it has the side effects of timeouts.



>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932p4674990.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>

Re: TomEE blocks additional EJBD connections after first 10

Posted by "Niall.Emmerson" <Ni...@visitscotland.com>.
Romain Manni-Bucau wrote
>> More than that
>> though, it would mean that if we ever wanted to have more than 10 Tomcat
>> servers connecting to the same TomEE server then we'd have the same
>> problem,
>>
> 
> no you would configure the pool to be bigger

Surely that just delays the problem? Let's assume that I need 11 Tomcat's to
act as EJBD clients, I set the default core threads to 15 to give me a bit
of leeway to add more if I need to. A couple of months down the line,
something happens but I now need to have 16 EJBD clients. Now I have the
same problem again but I've got to restart the production instance of TomEE
in order to get it to pick up a new value for the number of core threads and
as a result everything that relies on that server has to be taken offline
while the restart occurs. I could configure the number of core threads to be
huge, 200 say, same as the maximum possible threads but then each deployment
of a client application will result in a new thread being created while the
old thread used by the same client before the new deployment will remain
parked in the TomEE's memory forever.


Romain Manni-Bucau wrote
>> the 11th Tomcat wouldn't be able to send requests (though strangely, if I
>> understand this right, if we had 20 Tomcat's connecting to the same TomEE
>> then the 11th would work because of the new thread). While 20 Tomcats
>> connecting to the same TomEE in production is probably unlikely, 11 might
>> not be outwith the bounds of possibility.
>>
>> Isn't this something that should be fixed in the OpenEJB code so that
>> this
>> sort of situation can't happen? Perhaps creating a new thread if one of
>> the
>> existing ones isn't available within a set time period or something?
>>
>>
> I don't really see a bug here since you dont rely on the built in
> config/setup

Sorry, I don't understand. How am I not using the built in config/setup when
all I've done is change some of the default config values? I actually
switched on EJBD by using the configuration values you yourself provided in 
another post
<http://tomee-openejb.979440.n4.nabble.com/EJB-socket-deamon-in-tomee-tp4658792p4658793.html> 
. I don't understand what situation the current code logic would be useful
in. Assuming default values, 10 clients are ok, 11-19 aren't and 20-200 are
but if you do have 20-200 then the last 9 won't work? If the solution is
just to increase the number of core threads then why have a queue? If I'm
not meant to use EJBD then why is it part of TomEE and won't the same
problem exist for standalone OpenEJB but only without the option to get
around this issue by using HTTP?

To me it seems that the logic for new connections should be: When a new
connection is made, create a new thread up until the number of core threads,
after that, if one of the core threads is free then use that for the next
new connection but if they're all in use, create a new thread and destroy
whichever thread becomes unused next so that you return to the number of
core threads when possible.


Anyway, I'll enquire about ways around the firewall issue with HTTP. It
seems a shame that the EJBD protocol support doesn't work out of the box
though.




--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932p4674990.html
Sent from the TomEE Users mailing list archive at Nabble.com.

Re: TomEE blocks additional EJBD connections after first 10

Posted by Romain Manni-Bucau <rm...@gmail.com>.
2015-05-21 13:41 GMT+02:00 Niall.Emmerson <Ni...@visitscotland.com>
:

> Using a different protocol in production and dev isn't ideal.


that's hwever quite common (SSL config etc)


> More than that
> though, it would mean that if we ever wanted to have more than 10 Tomcat
> servers connecting to the same TomEE server then we'd have the same
> problem,
>

no you would configure the pool to be bigger


> the 11th Tomcat wouldn't be able to send requests (though strangely, if I
> understand this right, if we had 20 Tomcat's connecting to the same TomEE
> then the 11th would work because of the new thread). While 20 Tomcats
> connecting to the same TomEE in production is probably unlikely, 11 might
> not be outwith the bounds of possibility.
>
> Isn't this something that should be fixed in the OpenEJB code so that this
> sort of situation can't happen? Perhaps creating a new thread if one of the
> existing ones isn't available within a set time period or something?
>
>
I don't really see a bug here since you dont rely on the built in
config/setup


>
> Romain Manni-Bucau wrote
> > well you can since tomcat can have multiple connectors with filtering or
> > just configuring tomee webapp to refuse some ip with the dedicated valve.
>
> We might have other stuff on the same box which shouldn't be accessed from
> the outside world either. I don't know, it's not my decision to make. If we
> do though, that means we can't block connections at Tomcat rather than the
> firewall. Basically it's not an ideal solution either.
>
>
> Romain Manni-Bucau wrote
> > 2015-05-21 12:06 GMT+02:00 Niall.Emmerson &lt;
>
> > Niall.Emmerson@
>
> > &gt;
> > :
> >
> >> Romain Manni-Bucau wrote
> >> > ok I see,
> >> >
> >> > if the connection is not closed the keep alive server class keeps it
> >> > connected which is intended.
> >>
> >> And then, after creating the core threads, TomEE won't process any more
> >> connections until the queue becomes full and a new thread is created.
> New
> >> clients (or Tomcats with reloaded contexts) then hang waiting for their
> >> requests to be pushed into that new thread which, if the queue doesn't
> >> become full, will never happen.
> >>
> >>
> >> Romain Manni-Bucau wrote
> >> > This surely also means you use openejb-ejbd "embedded' mode and not
> >> tomee
> >> > http layer. (ie you let openejb opening its own ejbd port). Why not
> >> using
> >> > tomcat integration which simply relies on http? Shouldnt change
> >> anything
> >> > for dev.
> >>
> >> Aye, I am. The idea for that was because I don't want the TomEE server
> to
> >> accept any traffic from the outside world then the firewall between it
> >> and
> >> the Tomcat server could be set to block all HTTP traffic. If I was to
> >> switch
> >> to HTTP mode then obviously I couldn't do that sort of filtering. I take
> >> it
> >> in HTTP mode the keep alive stuff isn't there or it can accept more than
> >> 10
> >> connections?
> >>
> >>
> >>
> > well you can since tomcat can have multiple connectors with filtering or
> > just configuring tomee webapp to refuse some ip with the dedicated valve.
> >
> > Also note in prod you dont redeploy normally so this only affects your
> dev
> > where http is fine.
> >
> >
> >
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932p4674950.html
> >> Sent from the TomEE Users mailing list archive at Nabble.com.
> >>
>
>
>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932p4674954.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>

Re: TomEE blocks additional EJBD connections after first 10

Posted by "Niall.Emmerson" <Ni...@visitscotland.com>.
Using a different protocol in production and dev isn't ideal. More than that
though, it would mean that if we ever wanted to have more than 10 Tomcat
servers connecting to the same TomEE server then we'd have the same problem,
the 11th Tomcat wouldn't be able to send requests (though strangely, if I
understand this right, if we had 20 Tomcat's connecting to the same TomEE
then the 11th would work because of the new thread). While 20 Tomcats
connecting to the same TomEE in production is probably unlikely, 11 might
not be outwith the bounds of possibility.

Isn't this something that should be fixed in the OpenEJB code so that this
sort of situation can't happen? Perhaps creating a new thread if one of the
existing ones isn't available within a set time period or something?


Romain Manni-Bucau wrote
> well you can since tomcat can have multiple connectors with filtering or 
> just configuring tomee webapp to refuse some ip with the dedicated valve. 

We might have other stuff on the same box which shouldn't be accessed from
the outside world either. I don't know, it's not my decision to make. If we
do though, that means we can't block connections at Tomcat rather than the
firewall. Basically it's not an ideal solution either.


Romain Manni-Bucau wrote
> 2015-05-21 12:06 GMT+02:00 Niall.Emmerson &lt;

> Niall.Emmerson@

> &gt;
> :
> 
>> Romain Manni-Bucau wrote
>> > ok I see,
>> >
>> > if the connection is not closed the keep alive server class keeps it
>> > connected which is intended.
>>
>> And then, after creating the core threads, TomEE won't process any more
>> connections until the queue becomes full and a new thread is created. New
>> clients (or Tomcats with reloaded contexts) then hang waiting for their
>> requests to be pushed into that new thread which, if the queue doesn't
>> become full, will never happen.
>>
>>
>> Romain Manni-Bucau wrote
>> > This surely also means you use openejb-ejbd "embedded' mode and not
>> tomee
>> > http layer. (ie you let openejb opening its own ejbd port). Why not
>> using
>> > tomcat integration which simply relies on http? Shouldnt change
>> anything
>> > for dev.
>>
>> Aye, I am. The idea for that was because I don't want the TomEE server to
>> accept any traffic from the outside world then the firewall between it
>> and
>> the Tomcat server could be set to block all HTTP traffic. If I was to
>> switch
>> to HTTP mode then obviously I couldn't do that sort of filtering. I take
>> it
>> in HTTP mode the keep alive stuff isn't there or it can accept more than
>> 10
>> connections?
>>
>>
>>
> well you can since tomcat can have multiple connectors with filtering or
> just configuring tomee webapp to refuse some ip with the dedicated valve.
> 
> Also note in prod you dont redeploy normally so this only affects your dev
> where http is fine.
> 
> 
> 
>>
>>
>>
>> --
>> View this message in context:
>> http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932p4674950.html
>> Sent from the TomEE Users mailing list archive at Nabble.com.
>>





--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932p4674954.html
Sent from the TomEE Users mailing list archive at Nabble.com.

Re: TomEE blocks additional EJBD connections after first 10

Posted by Romain Manni-Bucau <rm...@gmail.com>.
2015-05-21 12:06 GMT+02:00 Niall.Emmerson <Ni...@visitscotland.com>
:

> Romain Manni-Bucau wrote
> > ok I see,
> >
> > if the connection is not closed the keep alive server class keeps it
> > connected which is intended.
>
> And then, after creating the core threads, TomEE won't process any more
> connections until the queue becomes full and a new thread is created. New
> clients (or Tomcats with reloaded contexts) then hang waiting for their
> requests to be pushed into that new thread which, if the queue doesn't
> become full, will never happen.
>
>
> Romain Manni-Bucau wrote
> > This surely also means you use openejb-ejbd "embedded' mode and not tomee
> > http layer. (ie you let openejb opening its own ejbd port). Why not using
> > tomcat integration which simply relies on http? Shouldnt change anything
> > for dev.
>
> Aye, I am. The idea for that was because I don't want the TomEE server to
> accept any traffic from the outside world then the firewall between it and
> the Tomcat server could be set to block all HTTP traffic. If I was to
> switch
> to HTTP mode then obviously I couldn't do that sort of filtering. I take it
> in HTTP mode the keep alive stuff isn't there or it can accept more than 10
> connections?
>
>
>
well you can since tomcat can have multiple connectors with filtering or
just configuring tomee webapp to refuse some ip with the dedicated valve.

Also note in prod you dont redeploy normally so this only affects your dev
where http is fine.



>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932p4674950.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>

Re: TomEE blocks additional EJBD connections after first 10

Posted by "Niall.Emmerson" <Ni...@visitscotland.com>.
Romain Manni-Bucau wrote
> ok I see,
> 
> if the connection is not closed the keep alive server class keeps it
> connected which is intended.

And then, after creating the core threads, TomEE won't process any more
connections until the queue becomes full and a new thread is created. New
clients (or Tomcats with reloaded contexts) then hang waiting for their
requests to be pushed into that new thread which, if the queue doesn't
become full, will never happen.


Romain Manni-Bucau wrote
> This surely also means you use openejb-ejbd "embedded' mode and not tomee
> http layer. (ie you let openejb opening its own ejbd port). Why not using
> tomcat integration which simply relies on http? Shouldnt change anything
> for dev.

Aye, I am. The idea for that was because I don't want the TomEE server to
accept any traffic from the outside world then the firewall between it and
the Tomcat server could be set to block all HTTP traffic. If I was to switch
to HTTP mode then obviously I couldn't do that sort of filtering. I take it
in HTTP mode the keep alive stuff isn't there or it can accept more than 10
connections? 





--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932p4674950.html
Sent from the TomEE Users mailing list archive at Nabble.com.

Re: TomEE blocks additional EJBD connections after first 10

Posted by Romain Manni-Bucau <rm...@gmail.com>.
ok I see,

if the connection is not closed the keep alive server class keeps it
connected which is intended.

This surely also means you use openejb-ejbd "embedded' mode and not tomee
http layer. (ie you let openejb opening its own ejbd port). Why not using
tomcat integration which simply relies on http? Shouldnt change anything
for dev.




Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>

2015-05-21 11:24 GMT+02:00 Niall.Emmerson <Ni...@visitscotland.com>
:

> I did see that comment but, at least the way it seems in my case anyway, is
> the problem is that the 10 core threads don't handle 19 runnables, they
> handle 10 because the threads it creates block on socket read at
> KeepAliveServer:248 - seemingly waiting for data from client applications
> that may or may not ever be sent.
>
> As requested, I've attached my thread dumps for the TomEE server and then
> the Tomcat server.
> Thanks.
>
> tomcat_thread_dump.txt
> <
> http://tomee-openejb.979440.n4.nabble.com/file/n4674946/tomcat_thread_dump.txt
> >
> tomee_thread_dump.txt
> <
> http://tomee-openejb.979440.n4.nabble.com/file/n4674946/tomee_thread_dump.txt
> >
>
>
> Romain Manni-Bucau wrote
> > Hello
> >
> > ServicePool intention is commented:
> >
> > /**Defaults.
> >  * This suggests that 10 core threads should cope with up to 19
> > runnables (threads + queue, whereby queue = threads - 1).
> >  * Any more than 19 runnables will spawn a thread to cope if the
> > thread count is less than 150.
> >  * If 150 threads are processing runnables and the queue is full then
> > block and wait for
> >  * a slot for up to 10 seconds before rejecting the runnable.
> >  * If a thread remains idle for more than 1 minute then it will be
> > removed.
> >  */
> >
> >
> > That said I would be curious to get  a thread dump of you hanging
> > applicationto check why thread are hanging.
> >
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau &lt;https://twitter.com/rmannibucau&gt; |  Blog
> > &lt;http://rmannibucau.wordpress.com&gt; | Github
> > &lt;https://github.com/rmannibucau&gt; |
> > LinkedIn &lt;https://www.linkedin.com/in/rmannibucau&gt; | Tomitriber
> > &lt;http://www.tomitribe.com&gt;
> >
> > 2015-05-20 15:12 GMT+02:00 Niall.Emmerson &lt;
>
> > Niall.Emmerson@
>
> > &gt;
> > :
> >
> >> Greetings All
> >>
> >> I'm having a problem with TomEE (1.7.1) when using Tomcat (8) to connect
> >> to remote beans via EJBD which I'm beginning to think might be a bug in
> >> the
> >> OpenEJB code somewhere.
> >>
> >> I'm developing and deploying my Tomcat webapp in Eclipse and when I make
> >> a
> >> change the updated code is automatically deployed to the server and the
> >> context reloaded. The first time I start up the server everything works
> >> well and I'm able to call the EJBs on TomEE correctly. After making
> >> changes
> >> to code though, specifically, after the Tomcat servers' context restarts
> >> 10
> >> times, on the 11th time TomEE no longer responds to InitialContext
> >> lookups
> >> and the Tomcat application hangs (it blocks waiting for a response from
> >> the
> >> TomEE server at org.apache.openejb.client.ProtocolMetadata:101 in the
> >> openejb-client jar deployed along with the web app).
> >>
> >> I think I've narrowed the problem(s) down to the following.
> >>
> >> When the Tomcat context reloads/webapp restarts, the openejb-client
> jar's
> >> connection pool seems to leak into memory so that all the connections
> >> opened before the context reloaded remain open (but can't be used). I
> >> suspect org.apache.openejb.client.Client remains leaked in memory. When
> >> the
> >> Tomcat webapp does another lookup, another Client instance is created
> >> along
> >> with another connection.
> >>
> >> When a connection is made to TomEE by Tomcat, the TomEE ServicePool
> >> (org.apache.openejb.server) spins up a thread called "OpenEJB.ejbd.X"
> >> (where x is an integer). This thread then remains in the runnable state
> >> waiting for a socket read input at KeepAliveServer:248. When the client
> >> side connection has been leaked into the Tomcat memory this socket read
> >> will never return anything and so the thread remainings running. The
> >> TomEE
> >> ServicePool is making use of a LinkedBlockingQueue. According to the
> >> JavaDoc that means that after the Tomcat context has reloaded 10 times
> >> (the
> >> default ThreadPoolExecutor's threadCore value), the request for a new
> >> thread will go into the LinkedBlockingQueue and remain there until the
> >> queue is full, at which point an additional thread will be created. This
> >> means that after all the core threads are in use, no additional
> >> connections
> >> can be made to TomEE (unless so many are made as to overwhem the queue).
> >>
> >> Something seems wrong with this logic to me?
> >> Surely the TomEE ServicePool shouldn't be using a ThreadPoolExecutor
> with
> >> a queuing strategy that causes subsequent connection attempts to block
> >> indefinitely if the threads that it creates also block indefinitely? Or
> >> perhaps it's not meant to be the case that the thread it creates block
> at
> >> all?
> >> Also, presumably the client connection pool should terminate the
> >> connections if the context is reloaded?
> >>
> >> Am I missing something? Some sort of configuration value or something?
> >>
> >> --
> >> View this message in context:
> >>
> http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932.html
> >> Sent from the TomEE Users mailing list archive at Nabble.com.
>
>
>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932p4674946.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>

Re: TomEE blocks additional EJBD connections after first 10

Posted by "Niall.Emmerson" <Ni...@visitscotland.com>.
I did see that comment but, at least the way it seems in my case anyway, is
the problem is that the 10 core threads don't handle 19 runnables, they
handle 10 because the threads it creates block on socket read at
KeepAliveServer:248 - seemingly waiting for data from client applications
that may or may not ever be sent.

As requested, I've attached my thread dumps for the TomEE server and then
the Tomcat server.
Thanks.

tomcat_thread_dump.txt
<http://tomee-openejb.979440.n4.nabble.com/file/n4674946/tomcat_thread_dump.txt>  
tomee_thread_dump.txt
<http://tomee-openejb.979440.n4.nabble.com/file/n4674946/tomee_thread_dump.txt>  


Romain Manni-Bucau wrote
> Hello
> 
> ServicePool intention is commented:
> 
> /**Defaults.
>  * This suggests that 10 core threads should cope with up to 19
> runnables (threads + queue, whereby queue = threads - 1).
>  * Any more than 19 runnables will spawn a thread to cope if the
> thread count is less than 150.
>  * If 150 threads are processing runnables and the queue is full then
> block and wait for
>  * a slot for up to 10 seconds before rejecting the runnable.
>  * If a thread remains idle for more than 1 minute then it will be
> removed.
>  */
> 
> 
> That said I would be curious to get  a thread dump of you hanging
> applicationto check why thread are hanging.
> 
> 
> 
> Romain Manni-Bucau
> @rmannibucau &lt;https://twitter.com/rmannibucau&gt; |  Blog
> &lt;http://rmannibucau.wordpress.com&gt; | Github
> &lt;https://github.com/rmannibucau&gt; |
> LinkedIn &lt;https://www.linkedin.com/in/rmannibucau&gt; | Tomitriber
> &lt;http://www.tomitribe.com&gt;
> 
> 2015-05-20 15:12 GMT+02:00 Niall.Emmerson &lt;

> Niall.Emmerson@

> &gt;
> :
> 
>> Greetings All
>>
>> I'm having a problem with TomEE (1.7.1) when using Tomcat (8) to connect
>> to remote beans via EJBD which I'm beginning to think might be a bug in
>> the
>> OpenEJB code somewhere.
>>
>> I'm developing and deploying my Tomcat webapp in Eclipse and when I make
>> a
>> change the updated code is automatically deployed to the server and the
>> context reloaded. The first time I start up the server everything works
>> well and I'm able to call the EJBs on TomEE correctly. After making
>> changes
>> to code though, specifically, after the Tomcat servers' context restarts
>> 10
>> times, on the 11th time TomEE no longer responds to InitialContext
>> lookups
>> and the Tomcat application hangs (it blocks waiting for a response from
>> the
>> TomEE server at org.apache.openejb.client.ProtocolMetadata:101 in the
>> openejb-client jar deployed along with the web app).
>>
>> I think I've narrowed the problem(s) down to the following.
>>
>> When the Tomcat context reloads/webapp restarts, the openejb-client jar's
>> connection pool seems to leak into memory so that all the connections
>> opened before the context reloaded remain open (but can't be used). I
>> suspect org.apache.openejb.client.Client remains leaked in memory. When
>> the
>> Tomcat webapp does another lookup, another Client instance is created
>> along
>> with another connection.
>>
>> When a connection is made to TomEE by Tomcat, the TomEE ServicePool
>> (org.apache.openejb.server) spins up a thread called "OpenEJB.ejbd.X"
>> (where x is an integer). This thread then remains in the runnable state
>> waiting for a socket read input at KeepAliveServer:248. When the client
>> side connection has been leaked into the Tomcat memory this socket read
>> will never return anything and so the thread remainings running. The
>> TomEE
>> ServicePool is making use of a LinkedBlockingQueue. According to the
>> JavaDoc that means that after the Tomcat context has reloaded 10 times
>> (the
>> default ThreadPoolExecutor's threadCore value), the request for a new
>> thread will go into the LinkedBlockingQueue and remain there until the
>> queue is full, at which point an additional thread will be created. This
>> means that after all the core threads are in use, no additional
>> connections
>> can be made to TomEE (unless so many are made as to overwhem the queue).
>>
>> Something seems wrong with this logic to me?
>> Surely the TomEE ServicePool shouldn't be using a ThreadPoolExecutor with
>> a queuing strategy that causes subsequent connection attempts to block
>> indefinitely if the threads that it creates also block indefinitely? Or
>> perhaps it's not meant to be the case that the thread it creates block at
>> all?
>> Also, presumably the client connection pool should terminate the
>> connections if the context is reloaded?
>>
>> Am I missing something? Some sort of configuration value or something?
>>
>> --
>> View this message in context:
>> http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932.html
>> Sent from the TomEE Users mailing list archive at Nabble.com.





--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932p4674946.html
Sent from the TomEE Users mailing list archive at Nabble.com.

Re: TomEE blocks additional EJBD connections after first 10

Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hello

ServicePool intention is commented:

/**Defaults.
 * This suggests that 10 core threads should cope with up to 19
runnables (threads + queue, whereby queue = threads - 1).
 * Any more than 19 runnables will spawn a thread to cope if the
thread count is less than 150.
 * If 150 threads are processing runnables and the queue is full then
block and wait for
 * a slot for up to 10 seconds before rejecting the runnable.
 * If a thread remains idle for more than 1 minute then it will be removed.
 */


That said I would be curious to get  a thread dump of you hanging
applicationto check why thread are hanging.



Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>

2015-05-20 15:12 GMT+02:00 Niall.Emmerson <Ni...@visitscotland.com>
:

> Greetings All
>
> I'm having a problem with TomEE (1.7.1) when using Tomcat (8) to connect
> to remote beans via EJBD which I'm beginning to think might be a bug in the
> OpenEJB code somewhere.
>
> I'm developing and deploying my Tomcat webapp in Eclipse and when I make a
> change the updated code is automatically deployed to the server and the
> context reloaded. The first time I start up the server everything works
> well and I'm able to call the EJBs on TomEE correctly. After making changes
> to code though, specifically, after the Tomcat servers' context restarts 10
> times, on the 11th time TomEE no longer responds to InitialContext lookups
> and the Tomcat application hangs (it blocks waiting for a response from the
> TomEE server at org.apache.openejb.client.ProtocolMetadata:101 in the
> openejb-client jar deployed along with the web app).
>
> I think I've narrowed the problem(s) down to the following.
>
> When the Tomcat context reloads/webapp restarts, the openejb-client jar's
> connection pool seems to leak into memory so that all the connections
> opened before the context reloaded remain open (but can't be used). I
> suspect org.apache.openejb.client.Client remains leaked in memory. When the
> Tomcat webapp does another lookup, another Client instance is created along
> with another connection.
>
> When a connection is made to TomEE by Tomcat, the TomEE ServicePool
> (org.apache.openejb.server) spins up a thread called "OpenEJB.ejbd.X"
> (where x is an integer). This thread then remains in the runnable state
> waiting for a socket read input at KeepAliveServer:248. When the client
> side connection has been leaked into the Tomcat memory this socket read
> will never return anything and so the thread remainings running. The TomEE
> ServicePool is making use of a LinkedBlockingQueue. According to the
> JavaDoc that means that after the Tomcat context has reloaded 10 times (the
> default ThreadPoolExecutor's threadCore value), the request for a new
> thread will go into the LinkedBlockingQueue and remain there until the
> queue is full, at which point an additional thread will be created. This
> means that after all the core threads are in use, no additional connections
> can be made to TomEE (unless so many are made as to overwhem the queue).
>
> Something seems wrong with this logic to me?
> Surely the TomEE ServicePool shouldn't be using a ThreadPoolExecutor with
> a queuing strategy that causes subsequent connection attempts to block
> indefinitely if the threads that it creates also block indefinitely? Or
> perhaps it's not meant to be the case that the thread it creates block at
> all?
> Also, presumably the client connection pool should terminate the
> connections if the context is reloaded?
>
> Am I missing something? Some sort of configuration value or something?
>
> Any help/advice would be appreciated.
> For Visitor Information<http://www.visitscotland.com>
> For information on VisitScotland<http://www.visitscotland.org>
> For information on Business Tourism<http://www.conventionscotland.com>
> For information on EventScotland<http://www.EventScotland.org>
>
> 2015 is Scotland's Year of Food and Drink, celebrating our exceptional
> natural larder, the role food and drink plays in shaping our economic
> success and the people, landscapes and culture that make our food heritage
> so unique.
> We invite you to come and experience the Land of Food and Drink. On
> Twitter? [VisitScotland Twitter]  <https://twitter.com/VisitScotNews>
> #TasteScotland
> Get involved and share your #brilliantmoments [VisitScotland Facebook] <
> https://www.facebook.com/visitscotland>  [VisitScotland Twitter] <
> https://twitter.com/visitscotland>  [VisitScotland Instagram] <
> http://instagram.com/visitscotland>
> ________________________________
> Before you print think about the ENVIRONMENT
>
> This email and any attachments are private and confidential. If you have
> received this email in error, please immediately notify the sender and
> permanently remove the email from your system. If you receive this email in
> error, you must not use, copy or disclose any of the information contained
> in this e-mail or any attachments.
>
> Any views or opinions expressed in this email and any attachments are
> those of the sender and do not necessarily represent the views or opinions
> of VisitScotland. Please note that VisitScotland's email system (including
> the content of emails and attachments) is subject to monitoring by us for
> the purposes of security and training.
>
> Tha am post-d seo (agus faidhle sam bith na chois) d?omhair agus
> sochairichte. Mas ann le mearachd a tha thu air am post-d fhaighinn, cuir
> fios gun d?il chun neach a chur thugaibh e agus dubh ?s an
> teachdaireachd-sa sa bhad. Na d?an lethbhreac den fhiosrachadh, na
> foillsich e ann an d?igh sam bith, na d?an gn?omh ri a linn.
>
> Dh'fhaodadh beachdan pearsanta a bhith gan cur an c?ill sam post-d seo
> nach eil a' riochdachadh beachdan VisitScotland. Tha siostam post-dealain
> VisitScotland (an teachdaireachd sam post-dealain seo agus ceangalachain
> sam bith) ga mheasadh agus ga chl?radh gu tuaireamach airson adhbharan
> t?arainteachd agus tr?anadh.
>
> Tha VisitScotland a' comhairleachadh gu bheil sibh a' fosgladh puist-d
> agus cheangalachain aig ur cunnart fh?in.
>
>
>
>
> --
> View this message in context:
> http://tomee-openejb.979440.n4.nabble.com/TomEE-blocks-additional-EJBD-connections-after-first-10-tp4674932.html
> Sent from the TomEE Users mailing list archive at Nabble.com.