You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@esme.apache.org by Markus Kohler <ma...@gmail.com> on 2009/11/27 17:28:22 UTC
Memory allocation analysis Episode II
Hi all,
In short, We are down from 90Mbyte to *19Mbyte* for sending one message and
I believe it should be possible to get under 10Mbyte without too much
effort. With 10 Mbyte we would comply to the performance standard of some
major ERP vendor, but honestly I believe we should be able to do even better
;-)
Out of the 19Mbyte(maybe even only 11, it's bit hard to get reproducible
results) 9,9 are consumed by scala.xml.XML$.loadString(java.lang.String)
mainly because every time the routine is called, a new Parser is
instantiated. In addition javax.xml.parsers.SAXParserFactory.newInstance()
is called, which hits (unless that has changed in recent JDK's) the file
system to find out which XML parser is configured. *I think ideally this
should be fixed in the Scala sources. *
The issue with the message formatted 300 times when all users are logged on
still seems to be there, at least the GC log suggests that, but I guess
David will attack that soon. I will send around the profiling reports later
on.
Regards,
Markus
"The best way to predict the future is to invent it" -- Alan Kay
Re: Memory allocation analysis Episode II
Posted by Richard Hirsch <hi...@gmail.com>.
Added performance test results from 11-27 to the wiki:
http://cwiki.apache.org/ESME/performance-tests.html
On Sat, Nov 28, 2009 at 9:40 AM, Richard Hirsch <hi...@gmail.com> wrote:
> Excellent idea.
>
> Let me know if I can help.
>
> I also think this public discussion regarding lift/scala performance
> is critical for the acceptance of both in the enterprise.
>
> D.
>
>
>
> On Sat, Nov 28, 2009 at 9:00 AM, Markus Kohler <ma...@gmail.com> wrote:
>> Hi Richard,
>> taking this to the mailing list.
>> My plan was always to promote ESME to developers by providing them some
>> valuable information about how to optimize a real world Scala/Lift
>> application. If we promote the blog post a bit through dzone, reddit etc. we
>> should be able to get a few thousand page views on a single day. To be
>> valuable I would have to do a little bit more work, e.g. numbers for the
>> response time before and after would certainly something people would ask
>> for. It would also good to clarify the Scala loadXML issue, something I
>> wanted to do anyway soon.
>> A little introduction into memory allocation tracing would also make
>> sense.
>> I could work on this on Monday night, if nothing else is more important.
>> Any comments here?
>>
>> Regards,
>> Markus
>>
>> "The best way to predict the future is to invent it" -- Alan Kay
>>
>>
>> On Sat, Nov 28, 2009 at 7:10 AM, Richard Hirsch <hi...@gmail.com>wrote:
>>
>>> The bar chart is a great idea. Would be perfect for a blog. We can
>>> show we had a memory prob with a reference to a particular wiki page.
>>> Then describe the code change and then put a link to another wiki
>>> page. If you create the chart and do a very quick write-up, just
>>> blurbs. I'll blend everything and write the blog.
>>>
>>> Thanks.
>>>
>>> D.
>>>
>>> On Sat, Nov 28, 2009 at 12:30 AM, Markus Kohler <ma...@gmail.com>
>>> wrote:
>>> > Hi,
>>> > -ESME out of svn 27.11.2009 running on jetty default setting. Turned out
>>> to
>>> > be only 64 Mbyte
>>> > - logged all 300+x users
>>> > - GC graph 1. attachment yellow line indicates 323 users logged on only
>>> need
>>> > around 52Mbyte
>>> > -Still GC issues when sending one message, but not as bad as before
>>> > Restarted jetty and did a memory allocation tracing profiling session for
>>> > one message send; details see 2. attachment
>>> > -Overall 18.6 Mbyte allocated
>>> > - 9,9 Mbyte in scala.xml.XML$.loadString(java.lang.String) >50% overall,
>>> see
>>> > 3. attachment.
>>> >
>>> > Maybe we should just start to provide a simple bar chart.
>>> > eg. we went down from 90-100 to 18,6.
>>> > I would like to do a cpu profiling session for one message send, but ATM
>>> > have problems with my network. Intend to post the result to the scala dev
>>> > list. Maybe there's a workaround and we can use some other Class/method.
>>> >
>>> > Greetings,
>>> > Markus
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > "The best way to predict the future is to invent it" -- Alan Kay
>>> >
>>> >
>>> > On Fri, Nov 27, 2009 at 10:29 PM, Richard Hirsch <hi...@gmail.com>
>>> > wrote:
>>> >>
>>> >> @David - I don't know whether the Scala community is following these
>>> >> threads. Maybe, you can point them to Markus' work and ask them to
>>> >> take a look at the memory usage of "scala.xml.XML$.loadString".
>>> >>
>>> >> @Markus if you'll sen me the reports I'll post another wiki entry and
>>> >> ten blog about it.
>>> >>
>>> >>
>>> >> Thanks.
>>> >>
>>> >> D.
>>> >>
>>> >>
>>> >> On Fri, Nov 27, 2009 at 5:28 PM, Markus Kohler <markus.kohler@gmail.com
>>> >
>>> >> wrote:
>>> >> > Hi all,
>>> >> > In short, We are down from 90Mbyte to *19Mbyte* for sending one
>>> message
>>> >> > and
>>> >> > I believe it should be possible to get under 10Mbyte without too much
>>> >> > effort. With 10 Mbyte we would comply to the performance standard of
>>> >> > some
>>> >> > major ERP vendor, but honestly I believe we should be able to do even
>>> >> > better
>>> >> > ;-)
>>> >> >
>>> >> > Out of the 19Mbyte(maybe even only 11, it's bit hard to get
>>> reproducible
>>> >> > results) 9,9 are consumed by
>>> scala.xml.XML$.loadString(java.lang.String)
>>> >> > mainly because every time the routine is called, a new Parser is
>>> >> > instantiated. In addition
>>> >> > javax.xml.parsers.SAXParserFactory.newInstance()
>>> >> > is called, which hits (unless that has changed in recent JDK's) the
>>> >> > file
>>> >> > system to find out which XML parser is configured. *I think ideally
>>> >> > this
>>> >> > should be fixed in the Scala sources. *
>>> >> >
>>> >> > The issue with the message formatted 300 times when all users are
>>> logged
>>> >> > on
>>> >> > still seems to be there, at least the GC log suggests that, but I
>>> guess
>>> >> > David will attack that soon. I will send around the profiling reports
>>> >> > later
>>> >> > on.
>>> >> >
>>> >> > Regards,
>>> >> > Markus
>>> >> >
>>> >> >
>>> >> > "The best way to predict the future is to invent it" -- Alan Kay
>>> >> >
>>> >
>>> >
>>>
>>
>
Re: Memory allocation analysis Episode II
Posted by Richard Hirsch <hi...@gmail.com>.
Excellent idea.
Let me know if I can help.
I also think this public discussion regarding lift/scala performance
is critical for the acceptance of both in the enterprise.
D.
On Sat, Nov 28, 2009 at 9:00 AM, Markus Kohler <ma...@gmail.com> wrote:
> Hi Richard,
> taking this to the mailing list.
> My plan was always to promote ESME to developers by providing them some
> valuable information about how to optimize a real world Scala/Lift
> application. If we promote the blog post a bit through dzone, reddit etc. we
> should be able to get a few thousand page views on a single day. To be
> valuable I would have to do a little bit more work, e.g. numbers for the
> response time before and after would certainly something people would ask
> for. It would also good to clarify the Scala loadXML issue, something I
> wanted to do anyway soon.
> A little introduction into memory allocation tracing would also make
> sense.
> I could work on this on Monday night, if nothing else is more important.
> Any comments here?
>
> Regards,
> Markus
>
> "The best way to predict the future is to invent it" -- Alan Kay
>
>
> On Sat, Nov 28, 2009 at 7:10 AM, Richard Hirsch <hi...@gmail.com>wrote:
>
>> The bar chart is a great idea. Would be perfect for a blog. We can
>> show we had a memory prob with a reference to a particular wiki page.
>> Then describe the code change and then put a link to another wiki
>> page. If you create the chart and do a very quick write-up, just
>> blurbs. I'll blend everything and write the blog.
>>
>> Thanks.
>>
>> D.
>>
>> On Sat, Nov 28, 2009 at 12:30 AM, Markus Kohler <ma...@gmail.com>
>> wrote:
>> > Hi,
>> > -ESME out of svn 27.11.2009 running on jetty default setting. Turned out
>> to
>> > be only 64 Mbyte
>> > - logged all 300+x users
>> > - GC graph 1. attachment yellow line indicates 323 users logged on only
>> need
>> > around 52Mbyte
>> > -Still GC issues when sending one message, but not as bad as before
>> > Restarted jetty and did a memory allocation tracing profiling session for
>> > one message send; details see 2. attachment
>> > -Overall 18.6 Mbyte allocated
>> > - 9,9 Mbyte in scala.xml.XML$.loadString(java.lang.String) >50% overall,
>> see
>> > 3. attachment.
>> >
>> > Maybe we should just start to provide a simple bar chart.
>> > eg. we went down from 90-100 to 18,6.
>> > I would like to do a cpu profiling session for one message send, but ATM
>> > have problems with my network. Intend to post the result to the scala dev
>> > list. Maybe there's a workaround and we can use some other Class/method.
>> >
>> > Greetings,
>> > Markus
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > "The best way to predict the future is to invent it" -- Alan Kay
>> >
>> >
>> > On Fri, Nov 27, 2009 at 10:29 PM, Richard Hirsch <hi...@gmail.com>
>> > wrote:
>> >>
>> >> @David - I don't know whether the Scala community is following these
>> >> threads. Maybe, you can point them to Markus' work and ask them to
>> >> take a look at the memory usage of "scala.xml.XML$.loadString".
>> >>
>> >> @Markus if you'll sen me the reports I'll post another wiki entry and
>> >> ten blog about it.
>> >>
>> >>
>> >> Thanks.
>> >>
>> >> D.
>> >>
>> >>
>> >> On Fri, Nov 27, 2009 at 5:28 PM, Markus Kohler <markus.kohler@gmail.com
>> >
>> >> wrote:
>> >> > Hi all,
>> >> > In short, We are down from 90Mbyte to *19Mbyte* for sending one
>> message
>> >> > and
>> >> > I believe it should be possible to get under 10Mbyte without too much
>> >> > effort. With 10 Mbyte we would comply to the performance standard of
>> >> > some
>> >> > major ERP vendor, but honestly I believe we should be able to do even
>> >> > better
>> >> > ;-)
>> >> >
>> >> > Out of the 19Mbyte(maybe even only 11, it's bit hard to get
>> reproducible
>> >> > results) 9,9 are consumed by
>> scala.xml.XML$.loadString(java.lang.String)
>> >> > mainly because every time the routine is called, a new Parser is
>> >> > instantiated. In addition
>> >> > javax.xml.parsers.SAXParserFactory.newInstance()
>> >> > is called, which hits (unless that has changed in recent JDK's) the
>> >> > file
>> >> > system to find out which XML parser is configured. *I think ideally
>> >> > this
>> >> > should be fixed in the Scala sources. *
>> >> >
>> >> > The issue with the message formatted 300 times when all users are
>> logged
>> >> > on
>> >> > still seems to be there, at least the GC log suggests that, but I
>> guess
>> >> > David will attack that soon. I will send around the profiling reports
>> >> > later
>> >> > on.
>> >> >
>> >> > Regards,
>> >> > Markus
>> >> >
>> >> >
>> >> > "The best way to predict the future is to invent it" -- Alan Kay
>> >> >
>> >
>> >
>>
>
Re: Memory allocation analysis Episode II
Posted by Markus Kohler <ma...@gmail.com>.
Hi Richard,
taking this to the mailing list.
My plan was always to promote ESME to developers by providing them some
valuable information about how to optimize a real world Scala/Lift
application. If we promote the blog post a bit through dzone, reddit etc. we
should be able to get a few thousand page views on a single day. To be
valuable I would have to do a little bit more work, e.g. numbers for the
response time before and after would certainly something people would ask
for. It would also good to clarify the Scala loadXML issue, something I
wanted to do anyway soon.
A little introduction into memory allocation tracing would also make
sense.
I could work on this on Monday night, if nothing else is more important.
Any comments here?
Regards,
Markus
"The best way to predict the future is to invent it" -- Alan Kay
On Sat, Nov 28, 2009 at 7:10 AM, Richard Hirsch <hi...@gmail.com>wrote:
> The bar chart is a great idea. Would be perfect for a blog. We can
> show we had a memory prob with a reference to a particular wiki page.
> Then describe the code change and then put a link to another wiki
> page. If you create the chart and do a very quick write-up, just
> blurbs. I'll blend everything and write the blog.
>
> Thanks.
>
> D.
>
> On Sat, Nov 28, 2009 at 12:30 AM, Markus Kohler <ma...@gmail.com>
> wrote:
> > Hi,
> > -ESME out of svn 27.11.2009 running on jetty default setting. Turned out
> to
> > be only 64 Mbyte
> > - logged all 300+x users
> > - GC graph 1. attachment yellow line indicates 323 users logged on only
> need
> > around 52Mbyte
> > -Still GC issues when sending one message, but not as bad as before
> > Restarted jetty and did a memory allocation tracing profiling session for
> > one message send; details see 2. attachment
> > -Overall 18.6 Mbyte allocated
> > - 9,9 Mbyte in scala.xml.XML$.loadString(java.lang.String) >50% overall,
> see
> > 3. attachment.
> >
> > Maybe we should just start to provide a simple bar chart.
> > eg. we went down from 90-100 to 18,6.
> > I would like to do a cpu profiling session for one message send, but ATM
> > have problems with my network. Intend to post the result to the scala dev
> > list. Maybe there's a workaround and we can use some other Class/method.
> >
> > Greetings,
> > Markus
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > "The best way to predict the future is to invent it" -- Alan Kay
> >
> >
> > On Fri, Nov 27, 2009 at 10:29 PM, Richard Hirsch <hi...@gmail.com>
> > wrote:
> >>
> >> @David - I don't know whether the Scala community is following these
> >> threads. Maybe, you can point them to Markus' work and ask them to
> >> take a look at the memory usage of "scala.xml.XML$.loadString".
> >>
> >> @Markus if you'll sen me the reports I'll post another wiki entry and
> >> ten blog about it.
> >>
> >>
> >> Thanks.
> >>
> >> D.
> >>
> >>
> >> On Fri, Nov 27, 2009 at 5:28 PM, Markus Kohler <markus.kohler@gmail.com
> >
> >> wrote:
> >> > Hi all,
> >> > In short, We are down from 90Mbyte to *19Mbyte* for sending one
> message
> >> > and
> >> > I believe it should be possible to get under 10Mbyte without too much
> >> > effort. With 10 Mbyte we would comply to the performance standard of
> >> > some
> >> > major ERP vendor, but honestly I believe we should be able to do even
> >> > better
> >> > ;-)
> >> >
> >> > Out of the 19Mbyte(maybe even only 11, it's bit hard to get
> reproducible
> >> > results) 9,9 are consumed by
> scala.xml.XML$.loadString(java.lang.String)
> >> > mainly because every time the routine is called, a new Parser is
> >> > instantiated. In addition
> >> > javax.xml.parsers.SAXParserFactory.newInstance()
> >> > is called, which hits (unless that has changed in recent JDK's) the
> >> > file
> >> > system to find out which XML parser is configured. *I think ideally
> >> > this
> >> > should be fixed in the Scala sources. *
> >> >
> >> > The issue with the message formatted 300 times when all users are
> logged
> >> > on
> >> > still seems to be there, at least the GC log suggests that, but I
> guess
> >> > David will attack that soon. I will send around the profiling reports
> >> > later
> >> > on.
> >> >
> >> > Regards,
> >> > Markus
> >> >
> >> >
> >> > "The best way to predict the future is to invent it" -- Alan Kay
> >> >
> >
> >
>
Re: Memory allocation analysis Episode II
Posted by Richard Hirsch <hi...@gmail.com>.
@David - I don't know whether the Scala community is following these
threads. Maybe, you can point them to Markus' work and ask them to
take a look at the memory usage of "scala.xml.XML$.loadString".
@Markus if you'll sen me the reports I'll post another wiki entry and
ten blog about it.
Thanks.
D.
On Fri, Nov 27, 2009 at 5:28 PM, Markus Kohler <ma...@gmail.com> wrote:
> Hi all,
> In short, We are down from 90Mbyte to *19Mbyte* for sending one message and
> I believe it should be possible to get under 10Mbyte without too much
> effort. With 10 Mbyte we would comply to the performance standard of some
> major ERP vendor, but honestly I believe we should be able to do even better
> ;-)
>
> Out of the 19Mbyte(maybe even only 11, it's bit hard to get reproducible
> results) 9,9 are consumed by scala.xml.XML$.loadString(java.lang.String)
> mainly because every time the routine is called, a new Parser is
> instantiated. In addition javax.xml.parsers.SAXParserFactory.newInstance()
> is called, which hits (unless that has changed in recent JDK's) the file
> system to find out which XML parser is configured. *I think ideally this
> should be fixed in the Scala sources. *
>
> The issue with the message formatted 300 times when all users are logged on
> still seems to be there, at least the GC log suggests that, but I guess
> David will attack that soon. I will send around the profiling reports later
> on.
>
> Regards,
> Markus
>
>
> "The best way to predict the future is to invent it" -- Alan Kay
>