You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@roller.apache.org by Dave Johnson <da...@nc.rr.com> on 2005/12/04 00:30:32 UTC
Roller 2.1-dev stability problems
Just a word of warning. Thursday, I deployed the latest code in the
Roller trunk to my site and since then my site's main page has gone
blank three times (I restored the page by clearing the site-wide page
cache) and the site has crashed twice (the JVM crashes, leaving no
stack trace). If you want to check out the new features you might want
to do it on a test system only. I'm not sure what's going on yet. I'll
be working on isolating the problem once I get back to work on Monday.
- Dave
Re: Roller 2.1-dev stability problems
Posted by Dave Johnson <da...@rollerweblogger.org>.
I'm running on latest SVN as of Friday morning.
Roller Weblogger 2.1-dev (20051209113230:dave)
I haven't seen a blank page since then and my "Page size < 50 bytes"
log message has not appeared in my logs since that deployment. We *may*
be in the clear, thanks to your exception handling changes.
- Dave
On Dec 10, 2005, at 10:20 AM, Allen Gilliland wrote:
> Dave,
>
> do you have any update on the blank page error that you were seeing on
> your site a while back? has the problem gone away? or are you still
> making tweaks to correct it? i haven't been able to reproduce it yet,
> but obviously if it's still an issue we need to figure out what's
> happening.
>
> -- Allen
>
>
> Dave Johnson wrote:
>
>>
>> On Dec 5, 2005, at 2:16 PM, Allen Gilliland wrote:
>>
>>> On Mon, 2005-12-05 at 09:51, Dave Johnson wrote:
>>>
>>>> On Dec 5, 2005, at 12:38 PM, Allen Gilliland wrote:
>>>>
>>>>> were you able to gather any more info about when the blank screen
>>>>> problem was happening on your site? like what the url was, what
>>>>> the
>>>>> cache key was, what the response code was, etc? It sounds like
>>>>> this is
>>>>> something wrong with how the request stack is being processed.
>>>>
>>>>
>>>> No, I wasn't able to figure out which URL was causing the problem.
>>>
>>>
>>> the problem was happening on the main page though, so /main.do right?
>>
>>
>> No, it was only happening on my blog page.
>>
>>
>>>> That may solve the problem. But, I thought we wanted to remove
>>>> the RequestFilter entirely?
>>>
>>> yes, that is definitely the ultimate goal.
>>>
>>>> Perhaps BasePageServlet should throw an exception and/or a
>>>> 404 when an invalid page template is requested.
>>>
>>>
>>> that sounds pretty reasonable. if we do a
>>> request.setAttribute("DisplayException", exception); in the
>>> BasePageServlet then that will be enough to allow the cache filters
>>> to
>>> know they should disregard anything from the cached response content.
>>> it would also make sense to send a 404 as well.
>>
>>
>> Why don't you do that and also comment out the RequestFilter entirely.
>> Or, I'll do it if you're busy doing something else.
>>
>> - Dave
>>
>
Re: Roller 2.1-dev stability problems
Posted by Allen Gilliland <Al...@Sun.COM>.
Dave,
do you have any update on the blank page error that you were seeing on
your site a while back? has the problem gone away? or are you still
making tweaks to correct it? i haven't been able to reproduce it yet,
but obviously if it's still an issue we need to figure out what's happening.
-- Allen
Dave Johnson wrote:
>
> On Dec 5, 2005, at 2:16 PM, Allen Gilliland wrote:
>
>> On Mon, 2005-12-05 at 09:51, Dave Johnson wrote:
>>
>>> On Dec 5, 2005, at 12:38 PM, Allen Gilliland wrote:
>>>
>>>> were you able to gather any more info about when the blank screen
>>>> problem was happening on your site? like what the url was, what the
>>>> cache key was, what the response code was, etc? It sounds like
>>>> this is
>>>> something wrong with how the request stack is being processed.
>>>
>>>
>>> No, I wasn't able to figure out which URL was causing the problem.
>>
>>
>> the problem was happening on the main page though, so /main.do right?
>
>
> No, it was only happening on my blog page.
>
>
>>> That may solve the problem. But, I thought we wanted to remove
>>> the RequestFilter entirely?
>>
>> yes, that is definitely the ultimate goal.
>>
>>> Perhaps BasePageServlet should throw an exception and/or a
>>> 404 when an invalid page template is requested.
>>
>>
>> that sounds pretty reasonable. if we do a
>> request.setAttribute("DisplayException", exception); in the
>> BasePageServlet then that will be enough to allow the cache filters to
>> know they should disregard anything from the cached response content.
>> it would also make sense to send a 404 as well.
>
>
> Why don't you do that and also comment out the RequestFilter entirely.
> Or, I'll do it if you're busy doing something else.
>
> - Dave
>
Re: Roller 2.1-dev stability problems
Posted by Dave Johnson <da...@rollerweblogger.org>.
On Dec 5, 2005, at 2:16 PM, Allen Gilliland wrote:
> On Mon, 2005-12-05 at 09:51, Dave Johnson wrote:
>> On Dec 5, 2005, at 12:38 PM, Allen Gilliland wrote:
>>> were you able to gather any more info about when the blank screen
>>> problem was happening on your site? like what the url was, what the
>>> cache key was, what the response code was, etc? It sounds like this
>>> is
>>> something wrong with how the request stack is being processed.
>>
>> No, I wasn't able to figure out which URL was causing the problem.
>
> the problem was happening on the main page though, so /main.do right?
No, it was only happening on my blog page.
>> That may solve the problem. But, I thought we wanted to remove
>> the RequestFilter entirely?
> yes, that is definitely the ultimate goal.
>
>> Perhaps BasePageServlet should throw an exception and/or a
>> 404 when an invalid page template is requested.
>
> that sounds pretty reasonable. if we do a
> request.setAttribute("DisplayException", exception); in the
> BasePageServlet then that will be enough to allow the cache filters to
> know they should disregard anything from the cached response content.
> it would also make sense to send a 404 as well.
Why don't you do that and also comment out the RequestFilter entirely.
Or, I'll do it if you're busy doing something else.
- Dave
Re: Roller 2.1-dev stability problems
Posted by Allen Gilliland <Al...@Sun.COM>.
On Mon, 2005-12-05 at 09:51, Dave Johnson wrote:
> On Dec 5, 2005, at 12:38 PM, Allen Gilliland wrote:
> > were you able to gather any more info about when the blank screen
> > problem was happening on your site? like what the url was, what the
> > cache key was, what the response code was, etc? It sounds like this is
> > something wrong with how the request stack is being processed.
>
> No, I wasn't able to figure out which URL was causing the problem.
the problem was happening on the main page though, so /main.do right?
>
>
> > If simply moving the RequestFilter back to its earlier position in the
> > filter chain has solved the problem then that should help us identify
> > where the problem lies, however I don't think that is the long term
> > solution. The RollerRequest object does a lot of talking to the db, so
> > we should really try and keep it where it is now, after the caching
> > filters.
>
> Yes, I too would like to figure out how to get RequestFilter out of the
> picture.
>
> I suspect that the problem is that we now catch template not found
> problems in BasePageServlet, which is behind cache filter, rather
> than the RequestFilter which is in front of the cache filter.
gotcha.
>
>
> > I've just commited a couple small changes which may help a little. The
> > most important one is a small modification to the RequestFilter so that
> > if it catches an exception while constructing the RollerRequest it now
> > sets the "DisplayException" request attribute so that the cache filters
> > know not to cache the result. This wasn't happening before because the
> > RequestFilter has traditionally been one of the first filters in the
> > chain.
>
> That may solve the problem. But, I thought we wanted to remove
> the RequestFilter entirely?
yes, that is definitely the ultimate goal.
>
> Perhaps BasePageServlet should throw an exception and/or a
> 404 when an invalid page template is requested.
that sounds pretty reasonable. if we do a
request.setAttribute("DisplayException", exception); in the
BasePageServlet then that will be enough to allow the cache filters to
know they should disregard anything from the cached response content.
it would also make sense to send a 404 as well.
-- Allen
>
> - Dave
>
Re: Roller 2.1-dev stability problems
Posted by Dave Johnson <da...@rollerweblogger.org>.
On Dec 5, 2005, at 12:38 PM, Allen Gilliland wrote:
> were you able to gather any more info about when the blank screen
> problem was happening on your site? like what the url was, what the
> cache key was, what the response code was, etc? It sounds like this is
> something wrong with how the request stack is being processed.
No, I wasn't able to figure out which URL was causing the problem.
> If simply moving the RequestFilter back to its earlier position in the
> filter chain has solved the problem then that should help us identify
> where the problem lies, however I don't think that is the long term
> solution. The RollerRequest object does a lot of talking to the db, so
> we should really try and keep it where it is now, after the caching
> filters.
Yes, I too would like to figure out how to get RequestFilter out of the
picture.
I suspect that the problem is that we now catch template not found
problems in BasePageServlet, which is behind cache filter, rather
than the RequestFilter which is in front of the cache filter.
> I've just commited a couple small changes which may help a little. The
> most important one is a small modification to the RequestFilter so that
> if it catches an exception while constructing the RollerRequest it now
> sets the "DisplayException" request attribute so that the cache filters
> know not to cache the result. This wasn't happening before because the
> RequestFilter has traditionally been one of the first filters in the
> chain.
That may solve the problem. But, I thought we wanted to remove
the RequestFilter entirely?
Perhaps BasePageServlet should throw an exception and/or a
404 when an invalid page template is requested.
- Dave
Re: Roller 2.1-dev stability problems
Posted by Allen Gilliland <Al...@Sun.COM>.
Dave,
were you able to gather any more info about when the blank screen
problem was happening on your site? like what the url was, what the
cache key was, what the response code was, etc? It sounds like this is
something wrong with how the request stack is being processed.
If simply moving the RequestFilter back to its earlier position in the
filter chain has solved the problem then that should help us identify
where the problem lies, however I don't think that is the long term
solution. The RollerRequest object does a lot of talking to the db, so
we should really try and keep it where it is now, after the caching
filters.
I've just commited a couple small changes which may help a little. The
most important one is a small modification to the RequestFilter so that
if it catches an exception while constructing the RollerRequest it now
sets the "DisplayException" request attribute so that the cache filters
know not to cache the result. This wasn't happening before because the
RequestFilter has traditionally been one of the first filters in the
chain.
I also still think we should not need the RequestFilter at all anymore.
In Roller 2.0 the RequestFilter did 3 things ...
1. create a RollerRequest object (parse and validate request, fetch data
from db)
2. rememberMe cookie logic
3. set updateTime request attribute if it wasn't set
#2 is gone now because we have Acegi. and I moved #3 into the
FlavorServlet because it only pertains to rss feeds anyways. This means
that right now the only thing the RequestFilter does is create a
RollerRequest object so it can be used later on, and I think we should
be fine simply letting our servlets do this rather than doing it in a
filter.
-- Allen
On Mon, 2005-12-05 at 05:43, Dave Johnson wrote:
> I added the RequestFilter back into the chain saturday and the blank
> screen problems have completely disappeared, but I did get a crash on
> Sunday. The JVM again disappeared without leaving any log messages.
> This is significant; I haven't see crashes like this before.
>
> - Dave
>
>
>
> On Dec 3, 2005, at 6:30 PM, Dave Johnson wrote:
>
> > Just a word of warning. Thursday, I deployed the latest code in the
> > Roller trunk to my site and since then my site's main page has gone
> > blank three times (I restored the page by clearing the site-wide page
> > cache) and the site has crashed twice (the JVM crashes, leaving no
> > stack trace). If you want to check out the new features you might want
> > to do it on a test system only. I'm not sure what's going on yet. I'll
> > be working on isolating the problem once I get back to work on Monday.
> >
> > - Dave
> >
>
Re: Roller 2.1-dev stability problems
Posted by Anil Gangolli <an...@busybuddha.org>.
No, just a typo in where you had put them. Just miscommunication. I
failed to communicate precisely what I was seeing earlier when I brought
it up,but I thought you understood and were already changing it, and I
saw some SVN commits go by that appeared to touch the stuff I was
talking about, so I left it. Anyway, hopefully, this is the only cause
of this.
In the event that we do still see the JVM just "going away," try to
ensure that the JVM process is running in a working directory where it
can write core dumps and log files and has no resource limit setting
preventing it. We should normally see a hotspot err (hs_err_<pid>.log)
file in the case that the JVM crashes. In my experience, though, these
are extremely rare in pure Java settings.
--a.
Dave Johnson wrote:
> Well, don't I feel like an ID10T now. I just committed a fix for that.
>
> Thanks Anil!
>
> - Dave
>
>
> On Dec 5, 2005, at 11:09 AM, Anil Gangolli wrote:
>
>>
>> Dave,
>>
>> I still see the System.exit() calls I had mentioned finding in the
>> run() (not main()) method of the TurnoverReferersTask and this is
>> likely causing mysterious exits.
>>
>> --a.
>>
>>
>> Dave Johnson wrote:
>>
>>> I added the RequestFilter back into the chain saturday and the blank
>>> screen problems have completely disappeared, but I did get a crash
>>> on Sunday. The JVM again disappeared without leaving any log
>>> messages. This is significant; I haven't see crashes like this before.
>>>
>>> - Dave
>>>
>>>
>>>
>>> On Dec 3, 2005, at 6:30 PM, Dave Johnson wrote:
>>>
>>>> Just a word of warning. Thursday, I deployed the latest code in the
>>>> Roller trunk to my site and since then my site's main page has gone
>>>> blank three times (I restored the page by clearing the site-wide
>>>> page cache) and the site has crashed twice (the JVM crashes,
>>>> leaving no stack trace). If you want to check out the new features
>>>> you might want to do it on a test system only. I'm not sure what's
>>>> going on yet. I'll be working on isolating the problem once I get
>>>> back to work on Monday.
>>>>
>>>> - Dave
>>>>
>>>
>>>
>>
>
>
Re: Roller 2.1-dev stability problems
Posted by Dave Johnson <da...@rollerweblogger.org>.
Well, don't I feel like an ID10T now. I just committed a fix for that.
Thanks Anil!
- Dave
On Dec 5, 2005, at 11:09 AM, Anil Gangolli wrote:
>
> Dave,
>
> I still see the System.exit() calls I had mentioned finding in the
> run() (not main()) method of the TurnoverReferersTask and this is
> likely causing mysterious exits.
>
> --a.
>
>
> Dave Johnson wrote:
>
>> I added the RequestFilter back into the chain saturday and the blank
>> screen problems have completely disappeared, but I did get a crash on
>> Sunday. The JVM again disappeared without leaving any log messages.
>> This is significant; I haven't see crashes like this before.
>>
>> - Dave
>>
>>
>>
>> On Dec 3, 2005, at 6:30 PM, Dave Johnson wrote:
>>
>>> Just a word of warning. Thursday, I deployed the latest code in the
>>> Roller trunk to my site and since then my site's main page has gone
>>> blank three times (I restored the page by clearing the site-wide
>>> page cache) and the site has crashed twice (the JVM crashes, leaving
>>> no stack trace). If you want to check out the new features you might
>>> want to do it on a test system only. I'm not sure what's going on
>>> yet. I'll be working on isolating the problem once I get back to
>>> work on Monday.
>>>
>>> - Dave
>>>
>>
>>
>
Re: Roller 2.1-dev stability problems
Posted by Anil Gangolli <an...@busybuddha.org>.
Dave,
I still see the System.exit() calls I had mentioned finding in the run()
(not main()) method of the TurnoverReferersTask and this is likely
causing mysterious exits.
--a.
Dave Johnson wrote:
> I added the RequestFilter back into the chain saturday and the blank
> screen problems have completely disappeared, but I did get a crash on
> Sunday. The JVM again disappeared without leaving any log messages.
> This is significant; I haven't see crashes like this before.
>
> - Dave
>
>
>
> On Dec 3, 2005, at 6:30 PM, Dave Johnson wrote:
>
>> Just a word of warning. Thursday, I deployed the latest code in the
>> Roller trunk to my site and since then my site's main page has gone
>> blank three times (I restored the page by clearing the site-wide page
>> cache) and the site has crashed twice (the JVM crashes, leaving no
>> stack trace). If you want to check out the new features you might
>> want to do it on a test system only. I'm not sure what's going on
>> yet. I'll be working on isolating the problem once I get back to work
>> on Monday.
>>
>> - Dave
>>
>
>
Re: Roller 2.1-dev stability problems
Posted by Dave Johnson <da...@rollerweblogger.org>.
I added the RequestFilter back into the chain saturday and the blank
screen problems have completely disappeared, but I did get a crash on
Sunday. The JVM again disappeared without leaving any log messages.
This is significant; I haven't see crashes like this before.
- Dave
On Dec 3, 2005, at 6:30 PM, Dave Johnson wrote:
> Just a word of warning. Thursday, I deployed the latest code in the
> Roller trunk to my site and since then my site's main page has gone
> blank three times (I restored the page by clearing the site-wide page
> cache) and the site has crashed twice (the JVM crashes, leaving no
> stack trace). If you want to check out the new features you might want
> to do it on a test system only. I'm not sure what's going on yet. I'll
> be working on isolating the problem once I get back to work on Monday.
>
> - Dave
>