JRoller Blows up with Roller 3.1

[Matthew Schmidt <ma...@dzone.com>]

Hi guys.  Since our upgrade to Roller 3.1, things have not been good in 
the land of JRoller.  The system will occasionally run for a few hours 
and then suddenly add nearly a gig of objects to the heap, all of which 
are instanly pushed to the old gen space and cannot be garbage 
collected.  Restarting the application server does not correct this 
problem, as it often comes right back immediately.  We can see nothing 
in the access logs that are different from when its running fine and 
then its suddenly not.  The search is currenly disabled, so that's not 
causing it. The referer processing is also disabled and our caches in 
Roller are not too large.  We've upgraded both Velocity and Hibernate to 
their latest released versions and that had no effect.  Looking into the 
heap dump, it seems that most of the space is in Strings, probably 
related to Velocity.  We've disabled velocity caching completely right 
now so see if that is the cause. 

Has anyone else seen anything like this?  We're running in a 1.5G heap 
and it just runs normally for a while then flakes out and then won't 
even operate normally after a restart for a while.  Eventually, enough 
restarts get is back into regular operation.  I'd appreciate any help 
you guys can give as without something, JRoller will likely plunge even 
further into dispair.

Thanks,
Matt

Re: Delete Entry have null value in entry title and entry id

[he...@gsa.gov]

Yes, Dave. This is in the 3.1 code base.

-hc





"Dave" <sn...@gmail.com> 
07/26/2007 11:01 AM
Please respond to
dev@roller.apache.org


To
dev@roller.apache.org
cc

Subject
Re: Delete Entry have null value in entry title and entry id






On 7/25/07, henry.chang@gsa.gov <he...@gsa.gov> wrote:
> Bug: When deleting an entry, the "Entry Title" and "Entry ID" are
> displayed as null value.
>
> The reason is web\WEB-INF\jsps\authoring\WeblogEntryRemove.jsp loads the
> form with session scope. Change the scope to request fixes the problem. 
I
> check the bug list and could not find anybody reported this bug. Do you
> know about this bug?

I wasn't aware of that. This is in 3.1 code base?

- Dave

Re: Delete Entry have null value in entry title and entry id

[Dave <sn...@gmail.com>]

On 7/25/07, henry.chang@gsa.gov <he...@gsa.gov> wrote:
> Bug: When deleting an entry, the "Entry Title" and "Entry ID" are
> displayed as null value.
>
> The reason is web\WEB-INF\jsps\authoring\WeblogEntryRemove.jsp loads the
> form with session scope. Change the scope to request fixes the problem. I
> check the bug list and could not find anybody reported this bug. Do you
> know about this bug?

I wasn't aware of that. This is in 3.1 code base?

- Dave

Re: JRoller Blows up with Roller 3.1

[Matthew Schmidt <ma...@dzone.com>]

Sorry, I meant to say Allen's instructions :)

-Matt

Matthew Schmidt wrote:
> Ohh, and I think we disabled the referer processing completely as per 
> your instructions earlier off list.
>
> Anil Gangolli wrote:
>> Can you get a heap image at a point when things have gone sour?
>>
>> Do you have async referrer processing enabled?
>>
>> If you are capturing access logs, can you see if there is an increase in
>> any bot/crawler activity at the times when the heap grows?
>>
>> --a.
>>
>> Matthew Schmidt wrote:
>>> Hi guys.  Since our upgrade to Roller 3.1, things have not been good 
>>> in the land of JRoller.  The system will occasionally run for a few 
>>> hours and then suddenly add nearly a gig of objects to the heap, all 
>>> of which are instanly pushed to the old gen space and cannot be 
>>> garbage collected.  Restarting the application server does not 
>>> correct this problem, as it often comes right back immediately.  We 
>>> can see nothing in the access logs that are different from when its 
>>> running fine and then its suddenly not.  The search is currenly 
>>> disabled, so that's not causing it. The referer processing is also 
>>> disabled and our caches in Roller are not too large.  We've upgraded 
>>> both Velocity and Hibernate to their latest released versions and 
>>> that had no effect.  Looking into the heap dump, it seems that most 
>>> of the space is in Strings, probably related to Velocity.  We've 
>>> disabled velocity caching completely right now so see if that is the 
>>> cause.
>>> Has anyone else seen anything like this?  We're running in a 1.5G 
>>> heap and it just runs normally for a while then flakes out and then 
>>> won't even operate normally after a restart for a while.  
>>> Eventually, enough restarts get is back into regular operation.  I'd 
>>> appreciate any help you guys can give as without something, JRoller 
>>> will likely plunge even further into dispair.
>>>
>>> Thanks,
>>> Matt
>>
>>

Delete Entry have null value in entry title and entry id

[he...@gsa.gov]

Bug: When deleting an entry, the "Entry Title" and "Entry ID" are 
displayed as null value.

The reason is web\WEB-INF\jsps\authoring\WeblogEntryRemove.jsp loads the 
form with session scope. Change the scope to request fixes the problem. I 
check the bug list and could not find anybody reported this bug. Do you 
know about this bug?

- hc

Re: JRoller Blows up with Roller 3.1

[Matthew Schmidt <ma...@dzone.com>]

Ohh, and I think we disabled the referer processing completely as per 
your instructions earlier off list.

Anil Gangolli wrote:
> Can you get a heap image at a point when things have gone sour?
>
> Do you have async referrer processing enabled?
>
> If you are capturing access logs, can you see if there is an increase in
> any bot/crawler activity at the times when the heap grows?
>
> --a.
>
> Matthew Schmidt wrote:
>> Hi guys.  Since our upgrade to Roller 3.1, things have not been good 
>> in the land of JRoller.  The system will occasionally run for a few 
>> hours and then suddenly add nearly a gig of objects to the heap, all 
>> of which are instanly pushed to the old gen space and cannot be 
>> garbage collected.  Restarting the application server does not 
>> correct this problem, as it often comes right back immediately.  We 
>> can see nothing in the access logs that are different from when its 
>> running fine and then its suddenly not.  The search is currenly 
>> disabled, so that's not causing it. The referer processing is also 
>> disabled and our caches in Roller are not too large.  We've upgraded 
>> both Velocity and Hibernate to their latest released versions and 
>> that had no effect.  Looking into the heap dump, it seems that most 
>> of the space is in Strings, probably related to Velocity.  We've 
>> disabled velocity caching completely right now so see if that is the 
>> cause.
>> Has anyone else seen anything like this?  We're running in a 1.5G 
>> heap and it just runs normally for a while then flakes out and then 
>> won't even operate normally after a restart for a while.  Eventually, 
>> enough restarts get is back into regular operation.  I'd appreciate 
>> any help you guys can give as without something, JRoller will likely 
>> plunge even further into dispair.
>>
>> Thanks,
>> Matt
>
>

Re: JRoller Blows up with Roller 3.1

[Allen Gilliland <al...@sun.com>]

Dave wrote:
> On 7/25/07, Allen Gilliland <al...@sun.com> wrote:
>> Without some information about what kind of traffic you are trying to
>> handle it is very tough to make recommendations.  I have experience with
>> our traffic for blogs.sun.com, but if you are getting more traffic then
>> us then any advice I can give is obviously less proven.
>>
>> IMHO you need more RAM for caching and possibly a full hardware refresh
>> including a second machine.  You said yourself that the heap fills up
>> with Strings, likely related to velocity and hence the rendering system.
>>   If this is the case then you need more RAM both for rendering and for
>> caching.  We can run BSC fine on a single machine (SunFire T2000) even
>> when restarting everything from scratch (we actually tested this
>> yesterday :( ), but we have a 3.6G jvm heap and 6G of additional memory
>> allocated to memcached for caching, so that could be the differentiating
>> factor here.
> 
> I agree that JRoller.com is probably light on hardware, RAM, number of
> Matts, etc. but what I'd like to understand is why this problem occurs
> with Roller 3.1 but was not happening with Roller 2.0. As I understand
> the situation, JRoller.com was running OK but not great before 3.1 and
> now is blowing up every X number of hours. I believe Matt is running
> with the very same hardware configuration, same database w/same
> content, same Servlet container, etc.
> 
> I wonder what has changed to cause this problem. Since the profiler
> seems to implicate Velocity, I looked at the way we used Velocity in
> 2.0 and the way we use it now and though we no longer use the
> VelocityServlet, we do use the same Velocity property settings, the
> same ResourceLoaders, etc. I can't see any significant difference.


That is definitely a good question, but a pretty large number of things 
changed between 2.x and 3.x, especially in the rendering code, so it's 
going to be hard to say exactly what changed to cause the problem.

It's also entirely possible that this was just the straw that broke the 
camel's back and this outcome was inevitable even without the upgrade. 
You said that the site was already suffering a bit before the upgrade, 
so that's already not a great sign.  We actually had a scenario like 
this happen to us on BSC for a day or two where all of a sudden one 
afternoon the site seemed to slow to a grinding halt.  It turned out 
that the problem was a mix of poor database queries by Roller and some 
inadequate database configuration work which resulted in a bunch of our 
queries becoming painfully slow once the data set had reached a certain 
size.

I don't have any answers though, it's a tough situation because you are 
mixing a lot of variables.  I still think hardware is the best place to 
start.

-- Allen


> 
> - Dave

Re: JRoller Blows up with Roller 3.1

[Dave <sn...@gmail.com>]

On 7/25/07, Allen Gilliland <al...@sun.com> wrote:
> Without some information about what kind of traffic you are trying to
> handle it is very tough to make recommendations.  I have experience with
> our traffic for blogs.sun.com, but if you are getting more traffic then
> us then any advice I can give is obviously less proven.
>
> IMHO you need more RAM for caching and possibly a full hardware refresh
> including a second machine.  You said yourself that the heap fills up
> with Strings, likely related to velocity and hence the rendering system.
>   If this is the case then you need more RAM both for rendering and for
> caching.  We can run BSC fine on a single machine (SunFire T2000) even
> when restarting everything from scratch (we actually tested this
> yesterday :( ), but we have a 3.6G jvm heap and 6G of additional memory
> allocated to memcached for caching, so that could be the differentiating
> factor here.

I agree that JRoller.com is probably light on hardware, RAM, number of
Matts, etc. but what I'd like to understand is why this problem occurs
with Roller 3.1 but was not happening with Roller 2.0. As I understand
the situation, JRoller.com was running OK but not great before 3.1 and
now is blowing up every X number of hours. I believe Matt is running
with the very same hardware configuration, same database w/same
content, same Servlet container, etc.

I wonder what has changed to cause this problem. Since the profiler
seems to implicate Velocity, I looked at the way we used Velocity in
2.0 and the way we use it now and though we no longer use the
VelocityServlet, we do use the same Velocity property settings, the
same ResourceLoaders, etc. I can't see any significant difference.

- Dave

Re: JRoller Blows up with Roller 3.1

[Allen Gilliland <al...@sun.com>]

Without some information about what kind of traffic you are trying to 
handle it is very tough to make recommendations.  I have experience with 
our traffic for blogs.sun.com, but if you are getting more traffic then 
us then any advice I can give is obviously less proven.

IMHO you need more RAM for caching and possibly a full hardware refresh 
including a second machine.  You said yourself that the heap fills up 
with Strings, likely related to velocity and hence the rendering system. 
  If this is the case then you need more RAM both for rendering and for 
caching.  We can run BSC fine on a single machine (SunFire T2000) even 
when restarting everything from scratch (we actually tested this 
yesterday :( ), but we have a 3.6G jvm heap and 6G of additional memory 
allocated to memcached for caching, so that could be the differentiating 
factor here.

One other thing to note which may be part of your memory problems.  The 
rendering system in Roller is designed to cache whole pages, and the 
only way to do that is to buffer the entire page while rendering 
happens.  This means that if you get a lot of traffic causing rendering 
at the same time then your heap can fill up with buffered output content 
(both pages & feeds).  So again, the reality here is that if you are 
running a large site then you need more than 1.5G of RAM, period.  I 
would be the first person to say that it's unrealistic to run a Roller 
site with 10K blogs on a single machine with only a 1.5G heap.

If you want you can completely disable the rendering caches by setting 
these properties in your config, although you're likely to run your cpu 
a bit crazy and still run out of memory ...

cache.sitewide.enabled=false
cache.weblogpage.enabled=false
cache.weblogfeed.enabled=false

-- Allen


Matthew Schmidt wrote:
> We have heap dumps both before and after it goes sour.  The primarily 
> point to a large increase in strings, with many references to velocity 
> tokens.  We didn't really notice any significant change in the traffic 
> when it happened.  It happens so fast that the system is quickly 
> non-responsive.
> 
> -Matt
> 
> Anil Gangolli wrote:
>> Can you get a heap image at a point when things have gone sour?
>>
>> Do you have async referrer processing enabled?
>>
>> If you are capturing access logs, can you see if there is an increase in
>> any bot/crawler activity at the times when the heap grows?
>>
>> --a.
>>
>> Matthew Schmidt wrote:
>>> Hi guys.  Since our upgrade to Roller 3.1, things have not been good 
>>> in the land of JRoller.  The system will occasionally run for a few 
>>> hours and then suddenly add nearly a gig of objects to the heap, all 
>>> of which are instanly pushed to the old gen space and cannot be 
>>> garbage collected.  Restarting the application server does not 
>>> correct this problem, as it often comes right back immediately.  We 
>>> can see nothing in the access logs that are different from when its 
>>> running fine and then its suddenly not.  The search is currenly 
>>> disabled, so that's not causing it. The referer processing is also 
>>> disabled and our caches in Roller are not too large.  We've upgraded 
>>> both Velocity and Hibernate to their latest released versions and 
>>> that had no effect.  Looking into the heap dump, it seems that most 
>>> of the space is in Strings, probably related to Velocity.  We've 
>>> disabled velocity caching completely right now so see if that is the 
>>> cause.
>>> Has anyone else seen anything like this?  We're running in a 1.5G 
>>> heap and it just runs normally for a while then flakes out and then 
>>> won't even operate normally after a restart for a while.  Eventually, 
>>> enough restarts get is back into regular operation.  I'd appreciate 
>>> any help you guys can give as without something, JRoller will likely 
>>> plunge even further into dispair.
>>>
>>> Thanks,
>>> Matt
>>
>>

Re: JRoller Blows up with Roller 3.1

[Matthew Schmidt <ma...@dzone.com>]

We have heap dumps both before and after it goes sour.  The primarily 
point to a large increase in strings, with many references to velocity 
tokens.  We didn't really notice any significant change in the traffic 
when it happened.  It happens so fast that the system is quickly 
non-responsive.

-Matt

Anil Gangolli wrote:
> Can you get a heap image at a point when things have gone sour?
>
> Do you have async referrer processing enabled?
>
> If you are capturing access logs, can you see if there is an increase in
> any bot/crawler activity at the times when the heap grows?
>
> --a.
>
> Matthew Schmidt wrote:
>> Hi guys.  Since our upgrade to Roller 3.1, things have not been good 
>> in the land of JRoller.  The system will occasionally run for a few 
>> hours and then suddenly add nearly a gig of objects to the heap, all 
>> of which are instanly pushed to the old gen space and cannot be 
>> garbage collected.  Restarting the application server does not 
>> correct this problem, as it often comes right back immediately.  We 
>> can see nothing in the access logs that are different from when its 
>> running fine and then its suddenly not.  The search is currenly 
>> disabled, so that's not causing it. The referer processing is also 
>> disabled and our caches in Roller are not too large.  We've upgraded 
>> both Velocity and Hibernate to their latest released versions and 
>> that had no effect.  Looking into the heap dump, it seems that most 
>> of the space is in Strings, probably related to Velocity.  We've 
>> disabled velocity caching completely right now so see if that is the 
>> cause.
>> Has anyone else seen anything like this?  We're running in a 1.5G 
>> heap and it just runs normally for a while then flakes out and then 
>> won't even operate normally after a restart for a while.  Eventually, 
>> enough restarts get is back into regular operation.  I'd appreciate 
>> any help you guys can give as without something, JRoller will likely 
>> plunge even further into dispair.
>>
>> Thanks,
>> Matt
>
>

Re: JRoller Blows up with Roller 3.1

[Anil Gangolli <an...@busybuddha.org>]

Can you get a heap image at a point when things have gone sour?

Do you have async referrer processing enabled?

If you are capturing access logs, can you see if there is an increase in
any bot/crawler activity at the times when the heap grows?

--a.

Matthew Schmidt wrote:
> Hi guys.  Since our upgrade to Roller 3.1, things have not been good 
> in the land of JRoller.  The system will occasionally run for a few 
> hours and then suddenly add nearly a gig of objects to the heap, all 
> of which are instanly pushed to the old gen space and cannot be 
> garbage collected.  Restarting the application server does not correct 
> this problem, as it often comes right back immediately.  We can see 
> nothing in the access logs that are different from when its running 
> fine and then its suddenly not.  The search is currenly disabled, so 
> that's not causing it. The referer processing is also disabled and our 
> caches in Roller are not too large.  We've upgraded both Velocity and 
> Hibernate to their latest released versions and that had no effect.  
> Looking into the heap dump, it seems that most of the space is in 
> Strings, probably related to Velocity.  We've disabled velocity 
> caching completely right now so see if that is the cause.
> Has anyone else seen anything like this?  We're running in a 1.5G heap 
> and it just runs normally for a while then flakes out and then won't 
> even operate normally after a restart for a while.  Eventually, enough 
> restarts get is back into regular operation.  I'd appreciate any help 
> you guys can give as without something, JRoller will likely plunge 
> even further into dispair.
>
> Thanks,
> Matt