You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jclouds.apache.org by Inbar Stolberg <in...@gmail.com> on 2014/05/28 08:05:18 UTC
jclouds cache size
hi all
i am working with a number of jclouds conexts and the problem is i see that
each one is using
cache (about 8mb if i am not mistaken)
i don't use the cache feature at all so i would very much like to reduce
this memory consumption to 0.
i have no problem doing this externally or internally in the jclouds code..
this is some of the memory heap .
as you can see we use a lot of Contexts so each ones counts.
the line below sows the size of a single ComputeServiceContextImpl in
itself not to big but it stacks up
Class Name
| Objects | Shallow Heap | Retained Heap
org.jclouds.compute.internal.ComputeServiceContextImpl | 470 |
15,040 | 626,228,176
org.jclouds.rest.internal.RestContextImpl | 2,351
| 94,040 | 501,228,920
org.jclouds.concurrent.internal.SyncProxy | 113,580
| 5,451,840 | 255,100,240
Class Name
| Shallow Heap | Retained Heap
org.jclouds.compute.internal.ComputeServiceContextImpl | 32 |
1,344,184
thanks in advance.
regards inbar
Re: jclouds cache size
Posted by Andrew Phillips <ap...@qrmedia.com>.
Hi Inbar
The thing is that you are probably "using the cache" without being
aware of it. Without further details as to the exact objects on the
heap I can only guess, but if you look at e.g. Reflection2 [1], you
will see that jclouds caches a lot of calls using reflection to
improve performance.
Perhaps a different question: is there any way to reduce the number of
contexts you are using, and so get the overall size down? Could you
give some more background as to *why* you have so many contexts?
Regards
ap
[1]
https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/reflect/Reflection2.java
Re: jclouds cache size
Posted by Andrew Phillips <an...@apache.org>.
> so what I am thinking is moving to latest 1.7
> and use merge some of the experimented branch.
> is the reference you sent is the only class I need to change or is there
> anything else?
From what I recall, there are caches in other places, but we'd need
to follow some of the reference chains to see which ones are actually
the ones that are heavily in use in your context.
As a first step, I'd recommend trying with "vanilla" 1.7.2 first and,
if the memory footprint is still too large, trying the patch we
discussed. If that doesn't help, we can always come back and look for
other options ;-)
ap
Re: jclouds cache size
Posted by Inbar Stolberg <in...@gmail.com>.
thanks Andrew.
so what I am thinking is moving to latest 1.7
and use merge some of the experimented branch.
is the reference you sent is the only class I need to change or is there
anything else?
and is there something i need to manually configure other than that?
thanks for the help
Inbar
On Wed, May 28, 2014 at 3:23 PM, Andrew Phillips <an...@apache.org> wrote:
> *java.util.LinkedHashMap$Entry |
>> 2,363 | * * 94,520*
>> *sun.reflect.generics.tree.SimpleClassTypeSignature | 3,826
>> |
>> 91,824*
>> *java.lang.Object[]
>> | 1,983 | 81,608*
>> *java.util.HashMap$Entry[]
>> |
>> 980 | 81,600*
>> *java.lang.reflect.Method
>> | 908 | 79,904*
>> *sun.reflect.generics.tree.TypeArgument[] |
>> 3,826 | 63,784*
>>
>
> We'd need to look at the reference chain in detail to be sure, but this
> data certainly appears consistent with some of the reflection-related
> caches in jclouds.
>
> As an experiment, you could try a custom build of jclouds that removes
> those caches from Reflection2 [1], just to see what the difference is, both
> in terms of performance and memory footprint..?
>
> ap
>
> [1] https://github.com/jclouds/jclouds/blob/master/core/src/
> main/java/org/jclouds/reflect/Reflection2.java
>
Re: jclouds cache size
Posted by Andrew Phillips <an...@apache.org>.
> *java.util.LinkedHashMap$Entry |
> 2,363 | * * 94,520*
> *sun.reflect.generics.tree.SimpleClassTypeSignature | 3,826 |
> 91,824*
> *java.lang.Object[]
> | 1,983 | 81,608*
> *java.util.HashMap$Entry[] |
> 980 | 81,600*
> *java.lang.reflect.Method
> | 908 | 79,904*
> *sun.reflect.generics.tree.TypeArgument[] |
> 3,826 | 63,784*
We'd need to look at the reference chain in detail to be sure, but
this data certainly appears consistent with some of the
reflection-related caches in jclouds.
As an experiment, you could try a custom build of jclouds that removes
those caches from Reflection2 [1], just to see what the difference is,
both in terms of performance and memory footprint..?
ap
[1]
https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/reflect/Reflection2.java
Re: jclouds cache size
Posted by Inbar Stolberg <in...@gmail.com>.
correct the version is very old. I will upgrade but i also see that this
will reduce about 250k-300k out of 1.3MB per computeContext but still too
large.
unfortunately i cannot reduce the amount of contexts (i am working with a
multi tenant openstack).
here is a deeper analysis of the context object:
char[]
| 4,253 | 162,080
java.lang.String
| 4,563 | 146,016
*java.util.LinkedHashMap$Entry |
2,363 | * * 94,520*
*sun.reflect.generics.tree.SimpleClassTypeSignature | 3,826 |
91,824*
*java.lang.Object[]
| 1,983 | 81,608*
*java.util.HashMap$Entry[] |
980 | 81,600*
*java.lang.reflect.Method
| 908 | 79,904*
*sun.reflect.generics.tree.TypeArgument[] |
3,826 | 63,784*
java.util.LinkedHashMap |
939 | 52,584
java.util.ArrayList
| 1,136 | 27,264
java.lang.reflect.Type[]
| 866 | 21,640
com.google.inject.spi.Dependency |
672 | 21,504
com.google.inject.Key |
869 | 20,856
com.google.inject.TypeLiteral |
859 | 20,616
java.util.concurrent.locks.ReentrantLock$NonfairSync | 518 |
16,576
com.google.inject.internal.util.$CustomConcurrentHashMap$Impl$Segment|
384 | 15,360
sun.reflect.annotation.AnnotationInvocationHandler | 571 |
13,704
sun.reflect.generics.tree.ClassTypeSignature | 848
| 13,568
java.util.HashMap$Entry |
408 | 13,056
java.lang.reflect.Constructor |
141 | 10,152
java.lang.reflect.Field
| 120 | 8,640
com.google.inject.internal.MoreTypes$ParameterizedTypeImpl |
359 | 8,616
com.google.inject.internal.SingleParameterInjector | 320
| 7,680
com.google.common.cache.LocalCache$Segment | 96 |
7,680
java.util.concurrent.atomic.AtomicReferenceArray | 480
| 7,680
sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl |
310 | 7,440
com.google.inject.internal.InternalFactoryToProviderAdapter |
306 | 7,344
com.google.inject.internal.util.$MapMaker$StrongEntry |
228 | 7,296
com.google.common.collect.RegularImmutableMap$TerminalEntry |
269 | 6,456
com.google.inject.internal.MembersInjectorImpl |
161 | 6,440
Total: 30 of 644 entries; 614 more |
42,024 | *1,344,184*
On Wed, May 28, 2014 at 9:18 AM, Andrew Gaul <ga...@apache.org> wrote:
> [moving thread to the jclouds user mailing list where it belongs.]
>
> Which version of jclouds do you use? We removed SyncProxy before
> jclouds 1.6 so you use a very old version! Please upgrade to 1.7.2 and
> test again.
>
> On Wed, May 28, 2014 at 09:05:18AM +0300, Inbar Stolberg wrote:
> > hi all
> > i am working with a number of jclouds conexts and the problem is i see
> that
> > each one is using
> > cache (about 8mb if i am not mistaken)
> > i don't use the cache feature at all so i would very much like to reduce
> > this memory consumption to 0.
> > i have no problem doing this externally or internally in the jclouds
> code..
> >
> > this is some of the memory heap .
> >
> > as you can see we use a lot of Contexts so each ones counts.
> >
> > the line below sows the size of a single ComputeServiceContextImpl in
> > itself not to big but it stacks up
> >
> >
> >
> >
> >
> > Class Name
> > | Objects | Shallow Heap | Retained Heap
> >
> > org.jclouds.compute.internal.ComputeServiceContextImpl | 470 |
> > 15,040 | 626,228,176
> >
> > org.jclouds.rest.internal.RestContextImpl |
> 2,351
> > | 94,040 | 501,228,920
> >
> > org.jclouds.concurrent.internal.SyncProxy | 113,580
> > | 5,451,840 | 255,100,240
> >
> >
> >
> > Class Name
> > | Shallow Heap | Retained Heap
> >
> > org.jclouds.compute.internal.ComputeServiceContextImpl | 32 |
> > 1,344,184
> >
> >
> >
> > thanks in advance.
> > regards inbar
>
> --
> Andrew Gaul
> http://gaul.org/
>
Re: jclouds cache size
Posted by Andrew Gaul <ga...@apache.org>.
[moving thread to the jclouds user mailing list where it belongs.]
Which version of jclouds do you use? We removed SyncProxy before
jclouds 1.6 so you use a very old version! Please upgrade to 1.7.2 and
test again.
On Wed, May 28, 2014 at 09:05:18AM +0300, Inbar Stolberg wrote:
> hi all
> i am working with a number of jclouds conexts and the problem is i see that
> each one is using
> cache (about 8mb if i am not mistaken)
> i don't use the cache feature at all so i would very much like to reduce
> this memory consumption to 0.
> i have no problem doing this externally or internally in the jclouds code..
>
> this is some of the memory heap .
>
> as you can see we use a lot of Contexts so each ones counts.
>
> the line below sows the size of a single ComputeServiceContextImpl in
> itself not to big but it stacks up
>
>
>
>
>
> Class Name
> | Objects | Shallow Heap | Retained Heap
>
> org.jclouds.compute.internal.ComputeServiceContextImpl | 470 |
> 15,040 | 626,228,176
>
> org.jclouds.rest.internal.RestContextImpl | 2,351
> | 94,040 | 501,228,920
>
> org.jclouds.concurrent.internal.SyncProxy | 113,580
> | 5,451,840 | 255,100,240
>
>
>
> Class Name
> | Shallow Heap | Retained Heap
>
> org.jclouds.compute.internal.ComputeServiceContextImpl | 32 |
> 1,344,184
>
>
>
> thanks in advance.
> regards inbar
--
Andrew Gaul
http://gaul.org/