You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomee.apache.org by Jlemoine <jl...@bcidaho.com> on 2016/10/03 12:44:08 UTC
Re: HTTP Errors on Load-Balanced TomEE Servers
Romain Manni-Bucau wrote
> Hi
>
> it would be interesting to tune the timeouts on the connector(s) -
> potentially on the LB too.
>
> If it happens again don't forget to take a thread dump of processes, it
> often helps to see what the server is doing and if it is just busy.
>
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://blog-rmannibucau.rhcloud.com> | Old Wordpress Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
> <http://www.tomitribe.com> | JavaEE Factory
> <https://javaeefactory-rmannibucau.rhcloud.com>
>
> 2016-09-29 21:25 GMT+02:00 Jlemoine <
> jlemoine@
> >:
>
>>
>>
>> --
>> View this message in context: http://tomee-openejb.979440.
>> n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-tp4680255.html
>> Sent from the TomEE Users mailing list archive at Nabble.com.
>>
Thanks Romain. I have been working on testing the connector timeouts,
although I haven't seen any performance improvements yet in my test
environment.
I can't mirror the same load in test as in prod but by throwing thousands of
sim users at the web servers I can create the same symptoms. Doing a thread
dump with VisualVM does seem to comfirm what I suspected... a socket issue
with the ajp connectors.
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-tp4680255p4680270.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: HTTP Errors on Load-Balanced TomEE Servers
Posted by Jlemoine <jl...@bcidaho.com>.
Romain Manni-Bucau wrote
> did you check your datasource pool too? default is pretty low and under
> load you can just wait for the DB depending the app
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://blog-rmannibucau.rhcloud.com> | Old Wordpress Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
> <http://www.tomitribe.com> | JavaEE Factory
> <https://javaeefactory-rmannibucau.rhcloud.com>
>
> 2016-10-03 17:11 GMT+02:00 Jlemoine <
> jlemoine@
> >:
No, I did not check the database connection pool settings. I will look into
that, thank you Romain.
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-tp4680255p4680277.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: HTTP Errors on Load-Balanced TomEE Servers
Posted by Romain Manni-Bucau <rm...@gmail.com>.
did you check your datasource pool too? default is pretty low and under
load you can just wait for the DB depending the app
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://blog-rmannibucau.rhcloud.com> | Old Wordpress Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com> | JavaEE Factory
<https://javaeefactory-rmannibucau.rhcloud.com>
2016-10-03 17:11 GMT+02:00 Jlemoine <jl...@bcidaho.com>:
> Romain Manni-Bucau wrote
> > 2016-10-03 15:43 GMT+02:00 Jlemoine <
>
> > jlemoine@
>
> > >:
> >
> >> 1. I'm not sure what a "very small timeout" would be. I currently have
> >> the
> >> ajp connector and LB timeouts at 20 seconds.
> >>
> >
> > 1ms?
> >
> >> 2. Yes, ajp does seem to keep threads open once created, which is where
> >> I
> >> think the problem was. No new sockets available to open. However, so
> >> far
> >> this has not happened on the TomEE side after I changed the LB method to
> >> Busyness and made the other changes to the ajp connectors that I
> >> mentioned
> >> in the first post.
> >>
> >>
> > this is weird, ajp perf boost comes normally from the binary protocol
> > usage
> > and connections staying open. Depends the setup for sure but if not used
> > wonder if you have any interest to use ajp.
> >
> >>
> >> --
> >> View this message in context: http://tomee-openejb.979440.
> >> n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-
> >> tp4680255p4680272.html
> >> Sent from the TomEE Users mailing list archive at Nabble.com.
> >>
>
> Yes Romain, from what I've researched what you're saying holds true. But I
> had the ajp attributes at default and this bizarre behavior from the LB
> Request method was totally overloading an instance.
> <http://tomee-openejb.979440.n4.nabble.com/file/n4680275/Image4.png>
>
>
>
>
>
>
>
>
> This isn't really load balancing for us and any end-users attached to that
> "a" instance were locked up and unable to process data back to the
> database.
> I'm pretty sure I have this ironed out now but I won't know unless the
> Business wants to try out my config changes in Production. Also, I'm still
> researching if there are other tuning options that I may need to try.
>
>
>
> --
> View this message in context: http://tomee-openejb.979440.
> n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-
> tp4680255p4680275.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>
Re: HTTP Errors on Load-Balanced TomEE Servers
Posted by Jlemoine <jl...@bcidaho.com>.
Romain Manni-Bucau wrote
> 2016-10-03 15:43 GMT+02:00 Jlemoine <
> jlemoine@
> >:
>
>> 1. I'm not sure what a "very small timeout" would be. I currently have
>> the
>> ajp connector and LB timeouts at 20 seconds.
>>
>
> 1ms?
>
>> 2. Yes, ajp does seem to keep threads open once created, which is where
>> I
>> think the problem was. No new sockets available to open. However, so
>> far
>> this has not happened on the TomEE side after I changed the LB method to
>> Busyness and made the other changes to the ajp connectors that I
>> mentioned
>> in the first post.
>>
>>
> this is weird, ajp perf boost comes normally from the binary protocol
> usage
> and connections staying open. Depends the setup for sure but if not used
> wonder if you have any interest to use ajp.
>
>>
>> --
>> View this message in context: http://tomee-openejb.979440.
>> n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-
>> tp4680255p4680272.html
>> Sent from the TomEE Users mailing list archive at Nabble.com.
>>
Yes Romain, from what I've researched what you're saying holds true. But I
had the ajp attributes at default and this bizarre behavior from the LB
Request method was totally overloading an instance.
<http://tomee-openejb.979440.n4.nabble.com/file/n4680275/Image4.png>
This isn't really load balancing for us and any end-users attached to that
"a" instance were locked up and unable to process data back to the database.
I'm pretty sure I have this ironed out now but I won't know unless the
Business wants to try out my config changes in Production. Also, I'm still
researching if there are other tuning options that I may need to try.
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-tp4680255p4680275.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: HTTP Errors on Load-Balanced TomEE Servers
Posted by Romain Manni-Bucau <rm...@gmail.com>.
2016-10-03 15:43 GMT+02:00 Jlemoine <jl...@bcidaho.com>:
> Romain Manni-Bucau wrote
> > 2016-10-03 14:44 GMT+02:00 Jlemoine <
>
> > jlemoine@
>
> > >:
> >
> >>
> > Don't get it wrong:
> >
> > 1. having a very small timeout should reject very fast clients, no perf
> > enhancements excepted you don't have too much waiting clients
> > 2. what do you mean by the socket issue? ajp is connected so keeps
> sockets
> > by design
> >
> >>
> >> --
> >> View this message in context: http://tomee-openejb.979440.
> >> n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-
> >> tp4680255p4680270.html
> >> Sent from the TomEE Users mailing list archive at Nabble.com.
> >>
>
> 1. I'm not sure what a "very small timeout" would be. I currently have
> the
> ajp connector and LB timeouts at 20 seconds.
>
1ms?
> 2. Yes, ajp does seem to keep threads open once created, which is where I
> think the problem was. No new sockets available to open. However, so far
> this has not happened on the TomEE side after I changed the LB method to
> Busyness and made the other changes to the ajp connectors that I mentioned
> in the first post.
>
>
this is weird, ajp perf boost comes normally from the binary protocol usage
and connections staying open. Depends the setup for sure but if not used
wonder if you have any interest to use ajp.
>
>
> --
> View this message in context: http://tomee-openejb.979440.
> n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-
> tp4680255p4680272.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>
Re: HTTP Errors on Load-Balanced TomEE Servers
Posted by Jlemoine <jl...@bcidaho.com>.
Romain Manni-Bucau wrote
> 2016-10-03 14:44 GMT+02:00 Jlemoine <
> jlemoine@
> >:
>
>>
> Don't get it wrong:
>
> 1. having a very small timeout should reject very fast clients, no perf
> enhancements excepted you don't have too much waiting clients
> 2. what do you mean by the socket issue? ajp is connected so keeps sockets
> by design
>
>>
>> --
>> View this message in context: http://tomee-openejb.979440.
>> n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-
>> tp4680255p4680270.html
>> Sent from the TomEE Users mailing list archive at Nabble.com.
>>
1. I'm not sure what a "very small timeout" would be. I currently have the
ajp connector and LB timeouts at 20 seconds.
2. Yes, ajp does seem to keep threads open once created, which is where I
think the problem was. No new sockets available to open. However, so far
this has not happened on the TomEE side after I changed the LB method to
Busyness and made the other changes to the ajp connectors that I mentioned
in the first post.
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-tp4680255p4680272.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: HTTP Errors on Load-Balanced TomEE Servers
Posted by Romain Manni-Bucau <rm...@gmail.com>.
2016-10-03 14:44 GMT+02:00 Jlemoine <jl...@bcidaho.com>:
> Romain Manni-Bucau wrote
> > Hi
> >
> > it would be interesting to tune the timeouts on the connector(s) -
> > potentially on the LB too.
> >
> > If it happens again don't forget to take a thread dump of processes, it
> > often helps to see what the server is doing and if it is just busy.
> >
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > <https://blog-rmannibucau.rhcloud.com> | Old Wordpress Blog
> > <http://rmannibucau.wordpress.com> | Github
> > <https://github.com/rmannibucau> |
> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
> > <http://www.tomitribe.com> | JavaEE Factory
> > <https://javaeefactory-rmannibucau.rhcloud.com>
> >
> > 2016-09-29 21:25 GMT+02:00 Jlemoine <
>
> > jlemoine@
>
> > >:
> >
> >>
> >>
> >> --
> >> View this message in context: http://tomee-openejb.979440.
> >> n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-tp4680255.html
> >> Sent from the TomEE Users mailing list archive at Nabble.com.
> >>
>
> Thanks Romain. I have been working on testing the connector timeouts,
> although I haven't seen any performance improvements yet in my test
> environment.
> I can't mirror the same load in test as in prod but by throwing thousands
> of
> sim users at the web servers I can create the same symptoms. Doing a
> thread
> dump with VisualVM does seem to comfirm what I suspected... a socket issue
> with the ajp connectors.
>
>
Don't get it wrong:
1. having a very small timeout should reject very fast clients, no perf
enhancements excepted you don't have too much waiting clients
2. what do you mean by the socket issue? ajp is connected so keeps sockets
by design
>
>
> --
> View this message in context: http://tomee-openejb.979440.
> n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-
> tp4680255p4680270.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
>