You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by Gaston Dombiak <ga...@jivesoftware.com> on 2007/06/20 02:54:35 UTC
Possible optimization?
Hey,
We are opening 60K concurrent connections to Openfire and generating
some heap dumps to analyze memory consumption. And we found something
that really surprised us since we never heard about it. A very big part
of the total memory is being consumed by ConcurrentHashMaps. The
ConcurrentHashMap uses internal objects to keep track of its content and
even if you have an "empty" ConcurrentHashMap the map will create as
many segments as its default size.
Tracing the references I see that BaseIoSession uses a ConcurrentHashMap
to keep track of the session attributes. Based on this finding I made
some changes to Openfire so that not highly concurrent code and heavily
used code would use synchronized + HashMap (the old style we can say)
instead of a ConcurrentHashMap. I don't see that MINA uses the session
attributes and in the Openfire's case the usage of those attributes is
high (but not that high) and never concurrent. So what do you think of
changing BaseIoSession to use a HashMap and use synchronized block when
accessing that variable? HashMap also uses internal objects but they are
smaller and less objects.
Regards,
-- Gato
Re: Possible optimization?
Posted by Mark Webb <el...@gmail.com>.
Trustin,
Not sure I understand the visibility issue. What do you mean? Are you
talking about visibility of method/fields or licensing?
As for Cliff's JavaOne presentation, take out the asterisks from the URL:
developers.sun.com/learning/javaoneonline/2007/pdf/TS-2862.pdf
On 6/20/07, Trustin Lee <tr...@gmail.com> wrote:
>
> On 6/20/07, 向秦贤 <fy...@gmail.com> wrote:
> > Hi,
> > Not HashMap, there is a cadicates.
> > http://blogs.azulsystems.com/cliff/2007/03/a_nonblocking_h.html
>
> Sounds interesting, but we need to address a possible visibility issue.
>
> > Its javaone speach:
> > developers.sun.com/learning/*javaone*online/2007/pdf/TS-2862.pdf
>
> It gives me 404 Not Found error. Could you give me the correct one?
>
> Thanks,
> Trustin
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
> --
> PGP Key ID: 0x0255ECA6
>
--
..Cheers
Mark
Re: Possible optimization?
Posted by Emmanuel Lecharny <el...@gmail.com>.
Hi,
this is a pretty interesting class for is, in Directory. I have done a
lot of benchmarks ths year, and what I get is that HashMap is heavily
used, and GC represnets around 30 % of all CPU, when the server is
really loaded, on a multi-CPU system.
So I guess that if we can have a fastest (maybe) but also a low memory
footprint lib, we will have a real gain.
It has to be checked, of course. btw, I checked the License, and it is
compatible with ASL2.
A good move, if the classes are ok, would be to push them in commons-collection.
Emmanuel
On 6/20/07, Julien Vermillard <jv...@archean.fr> wrote:
> On Wed, 20 Jun 2007 10:00:01 +0200
> "Dawie Malan" <da...@mxit.com> wrote:
>
> >
> > Trustin,
> >
> > You can also have a look at http://www.javolution.org/
> >
> > We're using it extensively, especially the FastXXX collections.
> >
> > Regards,
> >
> > Dawie Malan
> >
>
> I think it's more a matter of memory footprint than a speed
> issue.
>
> Julien
>
> >
> > -----Original Message-----
> > From: Trustin Lee [mailto:trustin@gmail.com]
> > Sent: 20 June 2007 09:48 AM
> > To: dev@mina.apache.org
> > Subject: Re: Possible optimization?
> >
> > On 6/20/07, 向秦贤 <fy...@gmail.com> wrote:
> > > Hi,
> > > Not HashMap, there is a cadicates.
> > > http://blogs.azulsystems.com/cliff/2007/03/a_nonblocking_h.html
> >
> > Sounds interesting, but we need to address a possible visibility
> > issue.
> >
> > > Its javaone speach:
> > > developers.sun.com/learning/*javaone*online/2007/pdf/TS-2862.pdf
> >
> > It gives me 404 Not Found error. Could you give me the correct one?
> >
> > Thanks,
> > Trustin
>
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Possible optimization?
Posted by 向秦贤 <fy...@gmail.com>.
Hi,
About javolution fast collections.
There is big strange issue, or problem(I can't and not want differ it), (of
cause not javolution self)
Sun seems to always reject javolution into java source code repository.
I merely get some not good conclusion about javolution but Only One from all
my known websites, whose name maybe long long after I remember.
And to myself, indeed a strange thing which I should dig into, it's that If
use db4o check out java object trees, you cannot see javolution objects from
db4o objectbase, and of cause you cannot search it.
Maybe jarcand of grizzly would check out it with grizzly, and jarand is
franch same as author of javolution.
Regards,
2007/6/20, Julien Vermillard <jv...@archean.fr>:
>
> On Wed, 20 Jun 2007 10:00:01 +0200
> "Dawie Malan" <da...@mxit.com> wrote:
>
> >
> > Trustin,
> >
> > You can also have a look at http://www.javolution.org/
> >
> > We're using it extensively, especially the FastXXX collections.
> >
> > Regards,
> >
> > Dawie Malan
> >
>
> I think it's more a matter of memory footprint than a speed
> issue.
>
> Julien
>
> >
> > -----Original Message-----
> > From: Trustin Lee [mailto:trustin@gmail.com]
> > Sent: 20 June 2007 09:48 AM
> > To: dev@mina.apache.org
> > Subject: Re: Possible optimization?
> >
> > On 6/20/07, 向秦贤 <fy...@gmail.com> wrote:
> > > Hi,
> > > Not HashMap, there is a cadicates.
> > > http://blogs.azulsystems.com/cliff/2007/03/a_nonblocking_h.html
> >
> > Sounds interesting, but we need to address a possible visibility
> > issue.
> >
> > > Its javaone speach:
> > > developers.sun.com/learning/*javaone*online/2007/pdf/TS-2862.pdf
> >
> > It gives me 404 Not Found error. Could you give me the correct one?
> >
> > Thanks,
> > Trustin
>
--
向秦贤
Re: Possible optimization?
Posted by Julien Vermillard <jv...@archean.fr>.
On Wed, 20 Jun 2007 10:00:01 +0200
"Dawie Malan" <da...@mxit.com> wrote:
>
> Trustin,
>
> You can also have a look at http://www.javolution.org/
>
> We're using it extensively, especially the FastXXX collections.
>
> Regards,
>
> Dawie Malan
>
I think it's more a matter of memory footprint than a speed
issue.
Julien
>
> -----Original Message-----
> From: Trustin Lee [mailto:trustin@gmail.com]
> Sent: 20 June 2007 09:48 AM
> To: dev@mina.apache.org
> Subject: Re: Possible optimization?
>
> On 6/20/07, 向秦贤 <fy...@gmail.com> wrote:
> > Hi,
> > Not HashMap, there is a cadicates.
> > http://blogs.azulsystems.com/cliff/2007/03/a_nonblocking_h.html
>
> Sounds interesting, but we need to address a possible visibility
> issue.
>
> > Its javaone speach:
> > developers.sun.com/learning/*javaone*online/2007/pdf/TS-2862.pdf
>
> It gives me 404 Not Found error. Could you give me the correct one?
>
> Thanks,
> Trustin
RE: Possible optimization?
Posted by Dawie Malan <da...@mxit.com>.
Trustin,
You can also have a look at http://www.javolution.org/
We're using it extensively, especially the FastXXX collections.
Regards,
Dawie Malan
-----Original Message-----
From: Trustin Lee [mailto:trustin@gmail.com]
Sent: 20 June 2007 09:48 AM
To: dev@mina.apache.org
Subject: Re: Possible optimization?
On 6/20/07, 向秦贤 <fy...@gmail.com> wrote:
> Hi,
> Not HashMap, there is a cadicates.
> http://blogs.azulsystems.com/cliff/2007/03/a_nonblocking_h.html
Sounds interesting, but we need to address a possible visibility issue.
> Its javaone speach:
> developers.sun.com/learning/*javaone*online/2007/pdf/TS-2862.pdf
It gives me 404 Not Found error. Could you give me the correct one?
Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
*******************************************************************
Click here to view our e-mail legal notice:
http://www.mxit.co.za/pdfs/mxit_legal.pdf or call: +27 21 888 7000
*******************************************************************
Re: Possible optimization?
Posted by Trustin Lee <tr...@gmail.com>.
On 6/20/07, 向秦贤 <fy...@gmail.com> wrote:
> Hi,
> Not HashMap, there is a cadicates.
> http://blogs.azulsystems.com/cliff/2007/03/a_nonblocking_h.html
Sounds interesting, but we need to address a possible visibility issue.
> Its javaone speach:
> developers.sun.com/learning/*javaone*online/2007/pdf/TS-2862.pdf
It gives me 404 Not Found error. Could you give me the correct one?
Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Possible optimization?
Posted by 向秦贤 <fy...@gmail.com>.
Hi,
Not HashMap, there is a cadicates.
http://blogs.azulsystems.com/cliff/2007/03/a_nonblocking_h.html
Its javaone speach:
developers.sun.com/learning/*javaone*online/2007/pdf/TS-2862.pdf
cliff also provide a library:
https://sourceforge.net/projects/high-scale-lib
2007/6/20, peter royal <pr...@apache.org>:
>
> On Jun 19, 2007, at 7:07 PM, Trustin Lee wrote:
> > WDYT? Shall we revert back to HashMap?
>
> Collections.synchronizedMap( new HashMap() ) sounds good :)
>
> -pete
>
> --
> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>
>
>
--
向秦贤
Re: Possible optimization?
Posted by peter royal <pr...@apache.org>.
On Jun 19, 2007, at 7:07 PM, Trustin Lee wrote:
> WDYT? Shall we revert back to HashMap?
Collections.synchronizedMap( new HashMap() ) sounds good :)
-pete
--
(peter.royal|osi)@pobox.com - http://fotap.org/~osi
Re: Possible optimization?
Posted by Trustin Lee <tr...@gmail.com>.
On 6/20/07, Mark Webb <el...@gmail.com> wrote:
> I think this is great information. I would like to see some test cases that
> support your information. In addition to the test cases, what
> environment/IDE are you performing this research on?
>
> Thanks and great work!
I think Gaston's comment is correct; ConcurrentHashMap takes up much
more memory than sychronized hashmap. We employed ConcurrentHashMap
for better concurrency, but we need balance between memory consumption
considering how concurrently the attributes will be accessed. I guess
most applications won't need concurrent hash map.
WDYT? Shall we revert back to HashMap?
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Possible optimization?
Posted by Mark Webb <el...@gmail.com>.
I think this is great information. I would like to see some test cases that
support your information. In addition to the test cases, what
environment/IDE are you performing this research on?
Thanks and great work!
On 6/19/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
>
> Hey,
>
>
>
> We are opening 60K concurrent connections to Openfire and generating
> some heap dumps to analyze memory consumption. And we found something
> that really surprised us since we never heard about it. A very big part
> of the total memory is being consumed by ConcurrentHashMaps. The
> ConcurrentHashMap uses internal objects to keep track of its content and
> even if you have an "empty" ConcurrentHashMap the map will create as
> many segments as its default size.
>
>
>
> Tracing the references I see that BaseIoSession uses a ConcurrentHashMap
> to keep track of the session attributes. Based on this finding I made
> some changes to Openfire so that not highly concurrent code and heavily
> used code would use synchronized + HashMap (the old style we can say)
> instead of a ConcurrentHashMap. I don't see that MINA uses the session
> attributes and in the Openfire's case the usage of those attributes is
> high (but not that high) and never concurrent. So what do you think of
> changing BaseIoSession to use a HashMap and use synchronized block when
> accessing that variable? HashMap also uses internal objects but they are
> smaller and less objects.
>
>
>
> Regards,
>
>
>
> -- Gato
>
>
--
..Cheers
Mark