You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@velocity.apache.org by Ryan Smith <ry...@authoria.com> on 2007/03/02 15:25:21 UTC
RE: Memory Footprint Help
I am going to start working on the answers to your questions. I've made
some headway, but I do have one question.
In looking closer at the cache, it seems that both the global cache and
the file cache both contain entries to the vm_global*_libraries. These
files are very large cache entries for us. Can you tell me which cache
they must be in? If they must only be in the global cache, how hard
would it be to take it out of the normal file cache?
Thanks
Ryan
-----Original Message-----
From: Will Glass-Husain [mailto:wglasshusain@gmail.com]
Sent: Monday, February 26, 2007 4:07 PM
To: Velocity Developers List
Subject: Re: Memory Footprint Help
Ryan,
To my knowledge we have not done significant work on memory profiling
with Velocity, this effort could be a big help.
Some possible questions to investigate:
* What classes are the memory hogs? The VM related files? Some of
the particular AST files?
* Which factors seem to tie up the most memory? Includes? Macros?
* Does the size of the context matter?
* Does the memory usage go up over time with continued compiling (e.g.
is there a growing leak)
WILL
On 2/26/07, Ryan Smith <ry...@authoria.com> wrote:
>
> I have posted the picture and attached it to bug
> (http://issues.apache.org/jira/browse/VELOCITY-223). The picture was
> there to simply illustrate the size of the velocity library file.
>
> I've done the same tests with the 1.5beta2 and the results are the
same
> although there is a noticeable difference in performance from a speed
> perspective. From a memory footprint perspective 1.5 is about 1%
larger
> than 1.4.
>
> The pages were continuously being compiled because they were not in
the
> cache since the cache size needed to be small to keep the memory
> footprint down. It wasn't the fault of any of the velocity code.
>
> At this time I am not that familiar with the Velocity code base but
any
> assistance in where to start looking would be greatly appreciated.
>
> Are there any places where velocity is hanging on to strings, tokens,
or
> processing instructions where we could potentially free them up?
>
> Are there other ways of factoring macros, files, or #parses that will
> help in reducing the footprint?
>
> Is there potentially extra data in any of the AST classes that isn't
> necessary after parsing or is potentially duplicate?
>
> I will do some more analysis of the memory, tokens, and directives and
> post the results to the JIRA bug.
>
> Thanks for your help.
>
> Ryan Smith
>
> -----Original Message-----
> From: Will Glass-Husain [mailto:wglasshusain@gmail.com]
> Sent: Saturday, February 24, 2007 7:47 PM
> To: Velocity Developers List
> Subject: Re: Memory Footprint Help
>
> Ryan,
>
> Unfortunately your picture was removed by the mail list software--
> perhaps you can raise a JIRA issue on this and attach the image and
> data?
>
> To answer your question, we would welcome assistance on this issue.
> If you have time and motivation, please dig into this. We'd be happy
> to help answer any questions on the code and/or offer ideas if not
> more direct help. If performance increases and the regression tests
> continue to pass we will almost certainly commit relevant changes.
>
> One note though -- there were several bug fixes related to caching,
> synchronization, introspection, and other subtle issues for Velocity
> 1.5. This version will be released in the next few days, but if you
> work with Velocity 1.5beta2 it is almost the same thing. Have you
> checked results from Velocity 1.5beta2?
>
> If the pages are continuously compiled that means the cache is not
> working, correct? How are you determining this?
>
> WILL
>
>
>
>
> On 2/24/07, Ryan Smith <ry...@authoria.com> wrote:
> >
> >
> >
> >
> >
> > We have a web site where we use velocity to generate our HTML pages.
> > Recently I was asked to help troubleshoot some performance issues
and
> the
> > root cause of our problem was that the velocity cache had grown to
> well over
> > 1GB in size causing the JVM to continuously GC to try to free up
> memory.
> >
> >
> >
> > As a short term fix I have grabbed the 1.5 beta ResourceCacheImpl
> class so
> > that a maximum cache size can be set and enforced. Unfortunately
when
> this
> > was done the performance of the site degraded significantly as the
> pages
> > were continuously compiled.
> >
> >
> >
> > I used the YourKit memory profiler and found the following
information
> about
> > the individual velocity cache entries (see attached picture):
> >
> > Name Cache Size File Size
> >
> > ---------------------------------------------------
> >
> > VM_framework_library.vm 9,596,472 130,500
> >
> > VM_buttons_library.vm 1,195,680 39,113
> >
> > VM_layout_library.vm 1,683,256 54,371
> >
> > admin/AdminHome.vm 32,505,168 979
> >
> > poNewGrid.vm 14,399,648 753
> >
> > poTemplateGrid.vm 14,369,000 774
> >
> > po/details.vm 11,140,952 8,368
> >
> > sub.vm 10,115,096 24,576
> >
> >
> >
> > At this time we have made a very heavy investment in velocity as our
> > presentation layer framework and love it. In order to meet our
> performance
> > goals, we need to keep as many of the velocity pages in cache as we
> can but
> > if we do that, we can only fit 2 web applications per Tomcat
> deployment in a
> > 32 bit environment.
> >
> > In searching through the JIRA issues, I found (
> > http://issues.apache.org/jira/browse/VELOCITY-450 ) and (
> > http://issues.apache.org/jira/browse/VELOCITY-223 ) that
> > reference this exact issue as well as the wiki entry talking about
how
> to
> > reduce the memory footprint.
> >
> > I am sending email to the developers list offering up my time to
> assist with
> > this issue. I was hoping that there would be someone who would be
> able to
> > point me in a direction to get me started and that there would
> potentially
> > be some "big wins" that we could take advantage of.
> >
> > Are there any places where velocity is hanging on to strings,
tokens,
> or
> > processing instructions where we could potentially free them up?
Are
> there
> > other ways of factoring macros, files, or #parses that will help in
> reducing
> > the footprint? Is there potentially extra data in any of the AST
> classes
> > that isn't necessary after parsing or is potentially duplicate?
> >
> >
> >
> > Thank you,
> >
> >
> >
> > Ryan Smith
> >
> >
> >
> > <<VelocityMemory.JPG>>
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@velocity.apache.org
> > For additional commands, e-mail: dev-help@velocity.apache.org
> >
>
>
> --
> Forio Business Simulations
>
> Will Glass-Husain
> wglass@forio.com
> www.forio.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@velocity.apache.org
> For additional commands, e-mail: dev-help@velocity.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@velocity.apache.org
> For additional commands, e-mail: dev-help@velocity.apache.org
>
>
--
Forio Business Simulations
Will Glass-Husain
wglass@forio.com
www.forio.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@velocity.apache.org
For additional commands, e-mail: dev-help@velocity.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@velocity.apache.org
For additional commands, e-mail: dev-help@velocity.apache.org
Re: Memory Footprint Help
Posted by Will Glass-Husain <wg...@gmail.com>.
Hi Rick,
No quick answer-- been a while since I've dug into this. (perhaps
someone else has a faster response).
Let me poke around with this over the weekend.
WILL
On 3/2/07, Ryan Smith <ry...@authoria.com> wrote:
>
> I am going to start working on the answers to your questions. I've made
> some headway, but I do have one question.
>
> In looking closer at the cache, it seems that both the global cache and
> the file cache both contain entries to the vm_global*_libraries. These
> files are very large cache entries for us. Can you tell me which cache
> they must be in? If they must only be in the global cache, how hard
> would it be to take it out of the normal file cache?
>
> Thanks
>
> Ryan
>
> -----Original Message-----
> From: Will Glass-Husain [mailto:wglasshusain@gmail.com]
> Sent: Monday, February 26, 2007 4:07 PM
> To: Velocity Developers List
> Subject: Re: Memory Footprint Help
>
> Ryan,
>
> To my knowledge we have not done significant work on memory profiling
> with Velocity, this effort could be a big help.
>
> Some possible questions to investigate:
>
> * What classes are the memory hogs? The VM related files? Some of
> the particular AST files?
>
> * Which factors seem to tie up the most memory? Includes? Macros?
>
> * Does the size of the context matter?
>
> * Does the memory usage go up over time with continued compiling (e.g.
> is there a growing leak)
>
> WILL
>
> On 2/26/07, Ryan Smith <ry...@authoria.com> wrote:
> >
> > I have posted the picture and attached it to bug
> > (http://issues.apache.org/jira/browse/VELOCITY-223). The picture was
> > there to simply illustrate the size of the velocity library file.
> >
> > I've done the same tests with the 1.5beta2 and the results are the
> same
> > although there is a noticeable difference in performance from a speed
> > perspective. From a memory footprint perspective 1.5 is about 1%
> larger
> > than 1.4.
> >
> > The pages were continuously being compiled because they were not in
> the
> > cache since the cache size needed to be small to keep the memory
> > footprint down. It wasn't the fault of any of the velocity code.
> >
> > At this time I am not that familiar with the Velocity code base but
> any
> > assistance in where to start looking would be greatly appreciated.
> >
> > Are there any places where velocity is hanging on to strings, tokens,
> or
> > processing instructions where we could potentially free them up?
> >
> > Are there other ways of factoring macros, files, or #parses that will
> > help in reducing the footprint?
> >
> > Is there potentially extra data in any of the AST classes that isn't
> > necessary after parsing or is potentially duplicate?
> >
> > I will do some more analysis of the memory, tokens, and directives and
> > post the results to the JIRA bug.
> >
> > Thanks for your help.
> >
> > Ryan Smith
> >
> > -----Original Message-----
> > From: Will Glass-Husain [mailto:wglasshusain@gmail.com]
> > Sent: Saturday, February 24, 2007 7:47 PM
> > To: Velocity Developers List
> > Subject: Re: Memory Footprint Help
> >
> > Ryan,
> >
> > Unfortunately your picture was removed by the mail list software--
> > perhaps you can raise a JIRA issue on this and attach the image and
> > data?
> >
> > To answer your question, we would welcome assistance on this issue.
> > If you have time and motivation, please dig into this. We'd be happy
> > to help answer any questions on the code and/or offer ideas if not
> > more direct help. If performance increases and the regression tests
> > continue to pass we will almost certainly commit relevant changes.
> >
> > One note though -- there were several bug fixes related to caching,
> > synchronization, introspection, and other subtle issues for Velocity
> > 1.5. This version will be released in the next few days, but if you
> > work with Velocity 1.5beta2 it is almost the same thing. Have you
> > checked results from Velocity 1.5beta2?
> >
> > If the pages are continuously compiled that means the cache is not
> > working, correct? How are you determining this?
> >
> > WILL
> >
> >
> >
> >
> > On 2/24/07, Ryan Smith <ry...@authoria.com> wrote:
> > >
> > >
> > >
> > >
> > >
> > > We have a web site where we use velocity to generate our HTML pages.
> > > Recently I was asked to help troubleshoot some performance issues
> and
> > the
> > > root cause of our problem was that the velocity cache had grown to
> > well over
> > > 1GB in size causing the JVM to continuously GC to try to free up
> > memory.
> > >
> > >
> > >
> > > As a short term fix I have grabbed the 1.5 beta ResourceCacheImpl
> > class so
> > > that a maximum cache size can be set and enforced. Unfortunately
> when
> > this
> > > was done the performance of the site degraded significantly as the
> > pages
> > > were continuously compiled.
> > >
> > >
> > >
> > > I used the YourKit memory profiler and found the following
> information
> > about
> > > the individual velocity cache entries (see attached picture):
> > >
> > > Name Cache Size File Size
> > >
> > > ---------------------------------------------------
> > >
> > > VM_framework_library.vm 9,596,472 130,500
> > >
> > > VM_buttons_library.vm 1,195,680 39,113
> > >
> > > VM_layout_library.vm 1,683,256 54,371
> > >
> > > admin/AdminHome.vm 32,505,168 979
> > >
> > > poNewGrid.vm 14,399,648 753
> > >
> > > poTemplateGrid.vm 14,369,000 774
> > >
> > > po/details.vm 11,140,952 8,368
> > >
> > > sub.vm 10,115,096 24,576
> > >
> > >
> > >
> > > At this time we have made a very heavy investment in velocity as our
> > > presentation layer framework and love it. In order to meet our
> > performance
> > > goals, we need to keep as many of the velocity pages in cache as we
> > can but
> > > if we do that, we can only fit 2 web applications per Tomcat
> > deployment in a
> > > 32 bit environment.
> > >
> > > In searching through the JIRA issues, I found (
> > > http://issues.apache.org/jira/browse/VELOCITY-450 ) and (
> > > http://issues.apache.org/jira/browse/VELOCITY-223 ) that
> > > reference this exact issue as well as the wiki entry talking about
> how
> > to
> > > reduce the memory footprint.
> > >
> > > I am sending email to the developers list offering up my time to
> > assist with
> > > this issue. I was hoping that there would be someone who would be
> > able to
> > > point me in a direction to get me started and that there would
> > potentially
> > > be some "big wins" that we could take advantage of.
> > >
> > > Are there any places where velocity is hanging on to strings,
> tokens,
> > or
> > > processing instructions where we could potentially free them up?
> Are
> > there
> > > other ways of factoring macros, files, or #parses that will help in
> > reducing
> > > the footprint? Is there potentially extra data in any of the AST
> > classes
> > > that isn't necessary after parsing or is potentially duplicate?
> > >
> > >
> > >
> > > Thank you,
> > >
> > >
> > >
> > > Ryan Smith
> > >
> > >
> > >
> > > <<VelocityMemory.JPG>>
> > >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@velocity.apache.org
> > > For additional commands, e-mail: dev-help@velocity.apache.org
> > >
> >
> >
> > --
> > Forio Business Simulations
> >
> > Will Glass-Husain
> > wglass@forio.com
> > www.forio.com
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@velocity.apache.org
> > For additional commands, e-mail: dev-help@velocity.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@velocity.apache.org
> > For additional commands, e-mail: dev-help@velocity.apache.org
> >
> >
>
>
> --
> Forio Business Simulations
>
> Will Glass-Husain
> wglass@forio.com
> www.forio.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@velocity.apache.org
> For additional commands, e-mail: dev-help@velocity.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@velocity.apache.org
> For additional commands, e-mail: dev-help@velocity.apache.org
>
>
--
Forio Business Simulations
Will Glass-Husain
wglass@forio.com
www.forio.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@velocity.apache.org
For additional commands, e-mail: dev-help@velocity.apache.org