You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tapestry.apache.org by Martin Grotzke <ma...@javakaffee.de> on 2008/05/10 02:39:47 UTC
T5 application not responding, blocked at
PerThreadServiceCreator.createObject
Hi,
yesterday we deployed our application upgraded from T5.0.5 to 5.0.10
(SNAPSHOT 2008-02-11) on the production system, now it hangs from time
to time with several thread blocked at
PerThreadServiceCreator.createObject.
We started with default settings for thetapestry.page-pool.hard-limit
and soft-limit (20, 5). With this we got a page pool exausted error on
one tomcat after some time, while others were fine with this.
Then we increased the soft-limit to 20 and the hard-limit to 100, which
also produced this error, but not always. Even without the pool exausted
error the application hung and did not respond for some minutes (about 3
minutes).
Now we have the hard-limit set to 150, which does not produce any pool
exausted errors, but still the application hangs.
All thread dumps that are created when the application is not responding
show blocked threads at PerThreadServiceCreator.createObject, and I
cannot see anything else that might cause this behaviour.
Looking at jconsole when the app is not responding shows an increasing
number of threads, and an increasing amount of used memory (e.g. from
500M to 800M), CPU is very low all the time.
I have uploaded a complete threaddump at http://senduit.com/a7afc7, an
aggregated version of this can be found at http://senduit.com/0d3f16.
We're running java-1.6.0_02, T5.0.10-SNAPSHOT and Tomcat 6.
Has anybody an idea what might be the reason for this behavior?
Thanx a lot for your help,
cheers,
Martin
Re: T5 application not responding, blocked at
PerThreadServiceCreator.createObject
Posted by Martin Grotzke <ma...@javakaffee.de>.
On Fri, 2008-05-09 at 17:47 -0700, Howard Lewis Ship wrote:
> The code in question was certainly changed by a bug fixed related to
> threading issues; rather than hangs, some users were seeing NPEs.
>
> Here's the bug: https://issues.apache.org/jira/browse/TAPESTRY-2141
I thought this would not apply to java 1.6, and we don't see NPEs in
ThreadLocal. Do you think the previous implementation of
PerThreadServiceCreator might cause the behaviour of our app?
>
> Fixed in 5.0.11.
Great! :)
>
> Can you try upgrading from 5.0.10-SNAPSHOT to 5.0.11?
Yes I can try, but this may take some time until we have this on
production. And having the current situation for two weeks on the
production system is not too far from beeing a desaster ;)
We also ran load tests before shifting the app with the changes to
production, but (unfortunately) didn't experience these problems.
Thanx for your help,
cheers,
Martin
>
> On Fri, May 9, 2008 at 5:39 PM, Martin Grotzke
> <ma...@javakaffee.de> wrote:
> > Hi,
> >
> > yesterday we deployed our application upgraded from T5.0.5 to 5.0.10
> > (SNAPSHOT 2008-02-11) on the production system, now it hangs from time
> > to time with several thread blocked at
> > PerThreadServiceCreator.createObject.
> >
> > We started with default settings for thetapestry.page-pool.hard-limit
> > and soft-limit (20, 5). With this we got a page pool exausted error on
> > one tomcat after some time, while others were fine with this.
> >
> > Then we increased the soft-limit to 20 and the hard-limit to 100, which
> > also produced this error, but not always. Even without the pool exausted
> > error the application hung and did not respond for some minutes (about 3
> > minutes).
> >
> > Now we have the hard-limit set to 150, which does not produce any pool
> > exausted errors, but still the application hangs.
> >
> > All thread dumps that are created when the application is not responding
> > show blocked threads at PerThreadServiceCreator.createObject, and I
> > cannot see anything else that might cause this behaviour.
> > Looking at jconsole when the app is not responding shows an increasing
> > number of threads, and an increasing amount of used memory (e.g. from
> > 500M to 800M), CPU is very low all the time.
> >
> > I have uploaded a complete threaddump at http://senduit.com/a7afc7, an
> > aggregated version of this can be found at http://senduit.com/0d3f16.
> >
> > We're running java-1.6.0_02, T5.0.10-SNAPSHOT and Tomcat 6.
> >
> > Has anybody an idea what might be the reason for this behavior?
> >
> > Thanx a lot for your help,
> > cheers,
> > Martin
> >
> >
> >
>
>
>
Re: T5 application not responding, blocked at PerThreadServiceCreator.createObject
Posted by Howard Lewis Ship <hl...@gmail.com>.
The code in question was certainly changed by a bug fixed related to
threading issues; rather than hangs, some users were seeing NPEs.
Here's the bug: https://issues.apache.org/jira/browse/TAPESTRY-2141
Fixed in 5.0.11.
Can you try upgrading from 5.0.10-SNAPSHOT to 5.0.11?
On Fri, May 9, 2008 at 5:39 PM, Martin Grotzke
<ma...@javakaffee.de> wrote:
> Hi,
>
> yesterday we deployed our application upgraded from T5.0.5 to 5.0.10
> (SNAPSHOT 2008-02-11) on the production system, now it hangs from time
> to time with several thread blocked at
> PerThreadServiceCreator.createObject.
>
> We started with default settings for thetapestry.page-pool.hard-limit
> and soft-limit (20, 5). With this we got a page pool exausted error on
> one tomcat after some time, while others were fine with this.
>
> Then we increased the soft-limit to 20 and the hard-limit to 100, which
> also produced this error, but not always. Even without the pool exausted
> error the application hung and did not respond for some minutes (about 3
> minutes).
>
> Now we have the hard-limit set to 150, which does not produce any pool
> exausted errors, but still the application hangs.
>
> All thread dumps that are created when the application is not responding
> show blocked threads at PerThreadServiceCreator.createObject, and I
> cannot see anything else that might cause this behaviour.
> Looking at jconsole when the app is not responding shows an increasing
> number of threads, and an increasing amount of used memory (e.g. from
> 500M to 800M), CPU is very low all the time.
>
> I have uploaded a complete threaddump at http://senduit.com/a7afc7, an
> aggregated version of this can be found at http://senduit.com/0d3f16.
>
> We're running java-1.6.0_02, T5.0.10-SNAPSHOT and Tomcat 6.
>
> Has anybody an idea what might be the reason for this behavior?
>
> Thanx a lot for your help,
> cheers,
> Martin
>
>
>
--
Howard M. Lewis Ship
Creator Apache Tapestry and Apache HiveMind
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: T5 application not responding, blocked at
PerThreadServiceCreator.createObject
Posted by Martin Grotzke <ma...@javakaffee.de>.
On Fri, 2008-05-09 at 18:08 -0700, Howard Lewis Ship wrote:
> Perhaps an upgrade to 5.0.11 rather than the .12-SNAPSHOT? I'm not
> sure how much changed between .10 and .11 API-wise.
I just changed the dependency to 5.0.11. Some changes have to be made,
but hopefully not too many. I'll have a look...
Cheers,
Martin
> It already seems
> like ancient history to me :-)
>
> On Fri, May 9, 2008 at 5:56 PM, Martin Grotzke
> <ma...@javakaffee.de> wrote:
> > On Fri, 2008-05-09 at 17:43 -0700, Howard Lewis Ship wrote:
> >> The problem with snapshots is that it makes it that much harder to
> >> figure out if this is a bug that's been fixed, or something new.
> > Completely true. I already search jira for fixes in 5.0.10 to 5.0.12
> > that might be related to this but found none.
> >
> > As there are changes in the API it's not that easy for us to upgrade to
> > 5.0.12, and IIRC there were issues with the calendar/date-component.
> > Then we also need to have testing cycles with the customers.
> > I'm also thinking about upgrading because then it's easier to track this
> > down, but it will take about 2 weeks until we could shift our
> > application with the latest T5 to production.
> >
> > That's why I asked directly on this list...
> >
> > Cheers,
> > Martin
> >
> >
> >>
> >> On Fri, May 9, 2008 at 5:39 PM, Martin Grotzke
> >> <ma...@javakaffee.de> wrote:
> >> > Hi,
> >> >
> >> > yesterday we deployed our application upgraded from T5.0.5 to 5.0.10
> >> > (SNAPSHOT 2008-02-11) on the production system, now it hangs from time
> >> > to time with several thread blocked at
> >> > PerThreadServiceCreator.createObject.
> >> >
> >> > We started with default settings for thetapestry.page-pool.hard-limit
> >> > and soft-limit (20, 5). With this we got a page pool exausted error on
> >> > one tomcat after some time, while others were fine with this.
> >> >
> >> > Then we increased the soft-limit to 20 and the hard-limit to 100, which
> >> > also produced this error, but not always. Even without the pool exausted
> >> > error the application hung and did not respond for some minutes (about 3
> >> > minutes).
> >> >
> >> > Now we have the hard-limit set to 150, which does not produce any pool
> >> > exausted errors, but still the application hangs.
> >> >
> >> > All thread dumps that are created when the application is not responding
> >> > show blocked threads at PerThreadServiceCreator.createObject, and I
> >> > cannot see anything else that might cause this behaviour.
> >> > Looking at jconsole when the app is not responding shows an increasing
> >> > number of threads, and an increasing amount of used memory (e.g. from
> >> > 500M to 800M), CPU is very low all the time.
> >> >
> >> > I have uploaded a complete threaddump at http://senduit.com/a7afc7, an
> >> > aggregated version of this can be found at http://senduit.com/0d3f16.
> >> >
> >> > We're running java-1.6.0_02, T5.0.10-SNAPSHOT and Tomcat 6.
> >> >
> >> > Has anybody an idea what might be the reason for this behavior?
> >> >
> >> > Thanx a lot for your help,
> >> > cheers,
> >> > Martin
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >
>
>
>
Re: T5 application not responding, blocked at
PerThreadServiceCreator.createObject
Posted by Martin Grotzke <ma...@javakaffee.de>.
Hi Howard,
I have upgraded our application to T5.0.11 and deployed it on one tomcat
for testing, this one is running fine now - the issue seems to be solved
with this. Thanx a lot! :)
Though there are still 2 issues introduced with the upgrade to 5.0.11:
- dateField component does not allow specifying format
(http://www.nabble.com/T5-dateField-component-does-not-allow-specifying-format-%28in-5.0.11%29-%5E-td17168397.html)
- setting initial sort order for grid in T5.0.11
(http://www.nabble.com/T5-setting-initial-sort-order-for-grid-in-T5.0.11-td17165163.html)
AFAICS the first one is represented by issue
TAPESTRY-2198 which is not yet assigned and has no target fix version.
Can you say what you think regarding this issue?
The second issue seems to be still open, I got no response until now
here on the list. Can you help me with this? Shall I enter an issue for
this one?
For us it's important to be able to tell our customers what's going on,
that we can give a timeline and provide orientation :)
Thanx for your help,
cheers,
Martin
On Fri, 2008-05-09 at 18:08 -0700, Howard Lewis Ship wrote:
> Perhaps an upgrade to 5.0.11 rather than the .12-SNAPSHOT? I'm not
> sure how much changed between .10 and .11 API-wise. It already seems
> like ancient history to me :-)
>
> On Fri, May 9, 2008 at 5:56 PM, Martin Grotzke
> <ma...@javakaffee.de> wrote:
> > On Fri, 2008-05-09 at 17:43 -0700, Howard Lewis Ship wrote:
> >> The problem with snapshots is that it makes it that much harder to
> >> figure out if this is a bug that's been fixed, or something new.
> > Completely true. I already search jira for fixes in 5.0.10 to 5.0.12
> > that might be related to this but found none.
> >
> > As there are changes in the API it's not that easy for us to upgrade to
> > 5.0.12, and IIRC there were issues with the calendar/date-component.
> > Then we also need to have testing cycles with the customers.
> > I'm also thinking about upgrading because then it's easier to track this
> > down, but it will take about 2 weeks until we could shift our
> > application with the latest T5 to production.
> >
> > That's why I asked directly on this list...
> >
> > Cheers,
> > Martin
> >
> >
> >>
> >> On Fri, May 9, 2008 at 5:39 PM, Martin Grotzke
> >> <ma...@javakaffee.de> wrote:
> >> > Hi,
> >> >
> >> > yesterday we deployed our application upgraded from T5.0.5 to 5.0.10
> >> > (SNAPSHOT 2008-02-11) on the production system, now it hangs from time
> >> > to time with several thread blocked at
> >> > PerThreadServiceCreator.createObject.
> >> >
> >> > We started with default settings for thetapestry.page-pool.hard-limit
> >> > and soft-limit (20, 5). With this we got a page pool exausted error on
> >> > one tomcat after some time, while others were fine with this.
> >> >
> >> > Then we increased the soft-limit to 20 and the hard-limit to 100, which
> >> > also produced this error, but not always. Even without the pool exausted
> >> > error the application hung and did not respond for some minutes (about 3
> >> > minutes).
> >> >
> >> > Now we have the hard-limit set to 150, which does not produce any pool
> >> > exausted errors, but still the application hangs.
> >> >
> >> > All thread dumps that are created when the application is not responding
> >> > show blocked threads at PerThreadServiceCreator.createObject, and I
> >> > cannot see anything else that might cause this behaviour.
> >> > Looking at jconsole when the app is not responding shows an increasing
> >> > number of threads, and an increasing amount of used memory (e.g. from
> >> > 500M to 800M), CPU is very low all the time.
> >> >
> >> > I have uploaded a complete threaddump at http://senduit.com/a7afc7, an
> >> > aggregated version of this can be found at http://senduit.com/0d3f16.
> >> >
> >> > We're running java-1.6.0_02, T5.0.10-SNAPSHOT and Tomcat 6.
> >> >
> >> > Has anybody an idea what might be the reason for this behavior?
> >> >
> >> > Thanx a lot for your help,
> >> > cheers,
> >> > Martin
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >
>
>
>
Re: T5 application not responding, blocked at PerThreadServiceCreator.createObject
Posted by Howard Lewis Ship <hl...@gmail.com>.
Perhaps an upgrade to 5.0.11 rather than the .12-SNAPSHOT? I'm not
sure how much changed between .10 and .11 API-wise. It already seems
like ancient history to me :-)
On Fri, May 9, 2008 at 5:56 PM, Martin Grotzke
<ma...@javakaffee.de> wrote:
> On Fri, 2008-05-09 at 17:43 -0700, Howard Lewis Ship wrote:
>> The problem with snapshots is that it makes it that much harder to
>> figure out if this is a bug that's been fixed, or something new.
> Completely true. I already search jira for fixes in 5.0.10 to 5.0.12
> that might be related to this but found none.
>
> As there are changes in the API it's not that easy for us to upgrade to
> 5.0.12, and IIRC there were issues with the calendar/date-component.
> Then we also need to have testing cycles with the customers.
> I'm also thinking about upgrading because then it's easier to track this
> down, but it will take about 2 weeks until we could shift our
> application with the latest T5 to production.
>
> That's why I asked directly on this list...
>
> Cheers,
> Martin
>
>
>>
>> On Fri, May 9, 2008 at 5:39 PM, Martin Grotzke
>> <ma...@javakaffee.de> wrote:
>> > Hi,
>> >
>> > yesterday we deployed our application upgraded from T5.0.5 to 5.0.10
>> > (SNAPSHOT 2008-02-11) on the production system, now it hangs from time
>> > to time with several thread blocked at
>> > PerThreadServiceCreator.createObject.
>> >
>> > We started with default settings for thetapestry.page-pool.hard-limit
>> > and soft-limit (20, 5). With this we got a page pool exausted error on
>> > one tomcat after some time, while others were fine with this.
>> >
>> > Then we increased the soft-limit to 20 and the hard-limit to 100, which
>> > also produced this error, but not always. Even without the pool exausted
>> > error the application hung and did not respond for some minutes (about 3
>> > minutes).
>> >
>> > Now we have the hard-limit set to 150, which does not produce any pool
>> > exausted errors, but still the application hangs.
>> >
>> > All thread dumps that are created when the application is not responding
>> > show blocked threads at PerThreadServiceCreator.createObject, and I
>> > cannot see anything else that might cause this behaviour.
>> > Looking at jconsole when the app is not responding shows an increasing
>> > number of threads, and an increasing amount of used memory (e.g. from
>> > 500M to 800M), CPU is very low all the time.
>> >
>> > I have uploaded a complete threaddump at http://senduit.com/a7afc7, an
>> > aggregated version of this can be found at http://senduit.com/0d3f16.
>> >
>> > We're running java-1.6.0_02, T5.0.10-SNAPSHOT and Tomcat 6.
>> >
>> > Has anybody an idea what might be the reason for this behavior?
>> >
>> > Thanx a lot for your help,
>> > cheers,
>> > Martin
>> >
>> >
>> >
>>
>>
>>
>
--
Howard M. Lewis Ship
Creator Apache Tapestry and Apache HiveMind
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: T5 application not responding, blocked at
PerThreadServiceCreator.createObject
Posted by Martin Grotzke <ma...@javakaffee.de>.
On Fri, 2008-05-09 at 17:43 -0700, Howard Lewis Ship wrote:
> The problem with snapshots is that it makes it that much harder to
> figure out if this is a bug that's been fixed, or something new.
Completely true. I already search jira for fixes in 5.0.10 to 5.0.12
that might be related to this but found none.
As there are changes in the API it's not that easy for us to upgrade to
5.0.12, and IIRC there were issues with the calendar/date-component.
Then we also need to have testing cycles with the customers.
I'm also thinking about upgrading because then it's easier to track this
down, but it will take about 2 weeks until we could shift our
application with the latest T5 to production.
That's why I asked directly on this list...
Cheers,
Martin
>
> On Fri, May 9, 2008 at 5:39 PM, Martin Grotzke
> <ma...@javakaffee.de> wrote:
> > Hi,
> >
> > yesterday we deployed our application upgraded from T5.0.5 to 5.0.10
> > (SNAPSHOT 2008-02-11) on the production system, now it hangs from time
> > to time with several thread blocked at
> > PerThreadServiceCreator.createObject.
> >
> > We started with default settings for thetapestry.page-pool.hard-limit
> > and soft-limit (20, 5). With this we got a page pool exausted error on
> > one tomcat after some time, while others were fine with this.
> >
> > Then we increased the soft-limit to 20 and the hard-limit to 100, which
> > also produced this error, but not always. Even without the pool exausted
> > error the application hung and did not respond for some minutes (about 3
> > minutes).
> >
> > Now we have the hard-limit set to 150, which does not produce any pool
> > exausted errors, but still the application hangs.
> >
> > All thread dumps that are created when the application is not responding
> > show blocked threads at PerThreadServiceCreator.createObject, and I
> > cannot see anything else that might cause this behaviour.
> > Looking at jconsole when the app is not responding shows an increasing
> > number of threads, and an increasing amount of used memory (e.g. from
> > 500M to 800M), CPU is very low all the time.
> >
> > I have uploaded a complete threaddump at http://senduit.com/a7afc7, an
> > aggregated version of this can be found at http://senduit.com/0d3f16.
> >
> > We're running java-1.6.0_02, T5.0.10-SNAPSHOT and Tomcat 6.
> >
> > Has anybody an idea what might be the reason for this behavior?
> >
> > Thanx a lot for your help,
> > cheers,
> > Martin
> >
> >
> >
>
>
>
Re: T5 application not responding, blocked at PerThreadServiceCreator.createObject
Posted by Howard Lewis Ship <hl...@gmail.com>.
The problem with snapshots is that it makes it that much harder to
figure out if this is a bug that's been fixed, or something new.
On Fri, May 9, 2008 at 5:39 PM, Martin Grotzke
<ma...@javakaffee.de> wrote:
> Hi,
>
> yesterday we deployed our application upgraded from T5.0.5 to 5.0.10
> (SNAPSHOT 2008-02-11) on the production system, now it hangs from time
> to time with several thread blocked at
> PerThreadServiceCreator.createObject.
>
> We started with default settings for thetapestry.page-pool.hard-limit
> and soft-limit (20, 5). With this we got a page pool exausted error on
> one tomcat after some time, while others were fine with this.
>
> Then we increased the soft-limit to 20 and the hard-limit to 100, which
> also produced this error, but not always. Even without the pool exausted
> error the application hung and did not respond for some minutes (about 3
> minutes).
>
> Now we have the hard-limit set to 150, which does not produce any pool
> exausted errors, but still the application hangs.
>
> All thread dumps that are created when the application is not responding
> show blocked threads at PerThreadServiceCreator.createObject, and I
> cannot see anything else that might cause this behaviour.
> Looking at jconsole when the app is not responding shows an increasing
> number of threads, and an increasing amount of used memory (e.g. from
> 500M to 800M), CPU is very low all the time.
>
> I have uploaded a complete threaddump at http://senduit.com/a7afc7, an
> aggregated version of this can be found at http://senduit.com/0d3f16.
>
> We're running java-1.6.0_02, T5.0.10-SNAPSHOT and Tomcat 6.
>
> Has anybody an idea what might be the reason for this behavior?
>
> Thanx a lot for your help,
> cheers,
> Martin
>
>
>
--
Howard M. Lewis Ship
Creator Apache Tapestry and Apache HiveMind
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org