You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by SandeepM <sk...@hotmail.com> on 2013/06/03 18:34:05 UTC

Solr 4.2.1 higher memory footprint vs Solr 3.5

Hi,

Using the same schema for both Solr 3.5 and Solr 4.2.1 and posting the same
data to both these server,  and the memory requirements seem to have gone up
sharply during request handling.
. Requests come in at around 200QPS.
. Document sizes are very large but that did not seem to be a problem with
3.5 (Lots of multivalued fields with large array lengths.)
Could you help me understand what change in SOLR 4.2.1 would attribute to
this higher memory requirement?

Also, in a different test, I ran a query to just get a list of all unique
ID's via a single query and no load and I see it complete in <500ms however
the time it takes to ship the data back to the client seems to be very
large.  Any idea what could be causing this behavior?

Would appreciate any help.

Regards,
-- Sandeep



--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-2-1-higher-memory-footprint-vs-Solr-3-5-tp4067879.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr 4.2.1 higher memory footprint vs Solr 3.5

Posted by adityab <ad...@yahoo.com>.
Hi All, 
I work with Sandeep M, so continued to his comments. We did observe a memory
growth. 
We use jdk1.6.0_45 with CMS. We see this issue because of large document
size. With large i mean our single document has large multivalued fields. 
We found that JIRA  LUCENE-4995
<https://issues.apache.org/jira/browse/LUCENE-4995>   is what we
experienced. and the patch seam to resolve our issue. We are performing more
test around it. 



--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-2-1-higher-memory-footprint-vs-Solr-3-5-tp4067879p4068886.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr 4.2.1 higher memory footprint vs Solr 3.5

Posted by Otis Gospodnetic <ot...@gmail.com>.
This is exactly what we did for a clients (alas using Elasticsearch). We
then observed better performance through SPM. We used the latest Oracle JVM.

Otis
Solr & ElasticSearch Support
http://sematext.com/
On Jun 7, 2013 2:55 AM, "Bernd Fehling" <be...@uni-bielefeld.de>
wrote:

> Hi Shawn,
>
> I also had CMS with tons of tuning options but still had once in a while
> bigger GC pause. After switching to JDK7 I tried G1GC with no other options
> and it runs perfekt.
> With CMS I saw that old and young generation where growing until they
> "had to do" a GC. This produces the sawtooth and also takes longer GC
> pause time.
> With G1GC the GC is more frequently and better timed, it is softer, more
> flexible.
> I just removed any old tuning and old GC and have only the G1GC option.
>
> ulimit -c unlimited
> ulimit -l 256
> ulimit -m unlimited
> ulimit -n 8192
> ulimit -s unlimited
> ulimit -v unlimited
>
> JAVA_OPTS="-server -d64 -Xmx20g -Xms20g -XX:+UseG1GC
>   -verbose:gc -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:gc.log"
>
> java version "1.7.0_07"
> Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
> Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>
> May be I have just luck with it, but for big heaps it works fine.
>
> Regards
> Bernd
>
>
> Am 06.06.2013 16:23, schrieb Shawn Heisey:
> > On 6/6/2013 3:50 AM, Bernd Fehling wrote:
> >> What helped me a lot was switching to G1GC.
> >> Faster, smoother, very little ripple, nearly no sawtooth.
> >
> > When I tried G1, it did indeed produce a better looking memory graph,
> > but it didn't do anything about my GC pauses.  They were several seconds
> > with just CMS and NewRatio, and they actually seemed to get slightly
> > worse when I tried G1 instead.
> >
> > To solve the GC pause problem, I've had to switch back to CMS and tack
> > on several more tuning options, most of which are CMS-specific.  I'm not
> > sure how to tune G1.  Have you done any additional tuning?
> >
> > Thanks,
> > Shawn
> >
>

Re: Solr 4.2.1 higher memory footprint vs Solr 3.5

Posted by Bernd Fehling <be...@uni-bielefeld.de>.
Hi Shawn,

I also had CMS with tons of tuning options but still had once in a while
bigger GC pause. After switching to JDK7 I tried G1GC with no other options
and it runs perfekt.
With CMS I saw that old and young generation where growing until they
"had to do" a GC. This produces the sawtooth and also takes longer GC pause time.
With G1GC the GC is more frequently and better timed, it is softer, more flexible.
I just removed any old tuning and old GC and have only the G1GC option.

ulimit -c unlimited
ulimit -l 256
ulimit -m unlimited
ulimit -n 8192
ulimit -s unlimited
ulimit -v unlimited

JAVA_OPTS="-server -d64 -Xmx20g -Xms20g -XX:+UseG1GC
  -verbose:gc -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:gc.log"

java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)

May be I have just luck with it, but for big heaps it works fine.

Regards
Bernd


Am 06.06.2013 16:23, schrieb Shawn Heisey:
> On 6/6/2013 3:50 AM, Bernd Fehling wrote:
>> What helped me a lot was switching to G1GC.
>> Faster, smoother, very little ripple, nearly no sawtooth.
> 
> When I tried G1, it did indeed produce a better looking memory graph,
> but it didn't do anything about my GC pauses.  They were several seconds
> with just CMS and NewRatio, and they actually seemed to get slightly
> worse when I tried G1 instead.
> 
> To solve the GC pause problem, I've had to switch back to CMS and tack
> on several more tuning options, most of which are CMS-specific.  I'm not
> sure how to tune G1.  Have you done any additional tuning?
> 
> Thanks,
> Shawn
> 

Re: Solr 4.2.1 higher memory footprint vs Solr 3.5

Posted by Shawn Heisey <so...@elyograg.org>.
On 6/6/2013 3:50 AM, Bernd Fehling wrote:
> What helped me a lot was switching to G1GC.
> Faster, smoother, very little ripple, nearly no sawtooth.

When I tried G1, it did indeed produce a better looking memory graph,
but it didn't do anything about my GC pauses.  They were several seconds
with just CMS and NewRatio, and they actually seemed to get slightly
worse when I tried G1 instead.

To solve the GC pause problem, I've had to switch back to CMS and tack
on several more tuning options, most of which are CMS-specific.  I'm not
sure how to tune G1.  Have you done any additional tuning?

Thanks,
Shawn


Re: Solr 4.2.1 higher memory footprint vs Solr 3.5

Posted by Bernd Fehling <be...@uni-bielefeld.de>.

Am 05.06.2013 18:09, schrieb SandeepM:
> /So we see the jagged edge waveform which keeps climbing (GC cycles don't
> completely collect memory over time).  Our test has a short capture from
> real traffic and we are replaying that via solrmeter./
> 
> Any idea why the memory climbs over time.  The GC should cleanup after data
> is shipped back.  Could there be a memory leak in SOLR?

Sorting can be a killer so take care about sorting only on fields prepared for sorting
and on fields which really make sense to sort.
The ultimate test/killer is sorting and faceting on doc id (the id field).
The problem is most times the fieldCache:
"Provides introspection of the Lucene FieldCache, this is **NOT** a cache that is managed by Solr."

Yes, the memory footprint right after start is higher for 4.2.1 than for 3.5, but
during runtime it is much lower with 4.2.1 (because of FST).

As for my system (45 mio. docs / 125 GB index) with 3.x I had 4 to 5 GB right after start
and 10 to 12 GB during runtime after several days. Now I have with 4.2.1 about 6 GB right
after start and are between 6.5 to 8 GB at runtime.
I don't see any increase in memory usage even after weeks, only peaks to 10 GB during replication
which will then decrease back to normal (6.5 to 8 GB).
So definately no memory leak.

What helped me a lot was switching to G1GC.
Faster, smoother, very little ripple, nearly no sawtooth.

Bernd

Re: Solr 4.2.1 higher memory footprint vs Solr 3.5

Posted by SandeepM <sk...@hotmail.com>.
/So we see the jagged edge waveform which keeps climbing (GC cycles don't
completely collect memory over time).  Our test has a short capture from
real traffic and we are replaying that via solrmeter./

Any idea why the memory climbs over time.  The GC should cleanup after data
is shipped back.  Could there be a memory leak in SOLR?

Appreciate any help.
Thanks.
-- Sandeep



--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-2-1-higher-memory-footprint-vs-Solr-3-5-tp4067879p4068378.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr 4.2.1 higher memory footprint vs Solr 3.5

Posted by Erick Erickson <er...@gmail.com>.
Shawn:

You're right, I thought I'd seen it as a <field> option but I think I
was confusing really old solr.

Thanks for catching, having gotten it wrong once I'm sure I'll
remember it better for next time!

Erick

On Tue, Jun 4, 2013 at 1:57 PM, SandeepM <sk...@hotmail.com> wrote:
> Thanks Eric and Shawn,
>
> Your explanations help understand where SOLR may be spending its time.
> Sounds like compression can be a CPU and heap hog. (I'll try to confirm this
> with the heapdumps)
>
> Initially we tried to keep the JVM heap sizes the same on both Solr 3.5 and
> 4.2.1, which was around 3GB ,which 3.5 handled well even with a 200QPS load.
> Moving to 4.2.1 with the same heap size instantly killed the Server.
> Changing the JVM to 6GB (double) did not help either.  We were seeing higher
> CPU and higher heap usage.
>
> We later changed cache settings so as to reduce their sizes, increased the
> JVM to 8GB and we see an improvement.  But over time, we do see that the
> Heap utilization slowly climbs as the 200QPS test is allowed to run, and
> sometimes leads to max heap being exceeded from the JConsole.  So we see the
> jagged edge waveform which keeps climbing (GC cycles don't completely
> collect memory over time).  Our test has a short capture from real traffic
> and we are replaying that via solrmeter.
>
> Thanks.
> Regards,
> -- Sandeep
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-2-1-higher-memory-footprint-vs-Solr-3-5-tp4067879p4068150.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr 4.2.1 higher memory footprint vs Solr 3.5

Posted by SandeepM <sk...@hotmail.com>.
Thanks Eric and Shawn,

Your explanations help understand where SOLR may be spending its time. 
Sounds like compression can be a CPU and heap hog. (I'll try to confirm this
with the heapdumps)

Initially we tried to keep the JVM heap sizes the same on both Solr 3.5 and
4.2.1, which was around 3GB ,which 3.5 handled well even with a 200QPS load. 
Moving to 4.2.1 with the same heap size instantly killed the Server. 
Changing the JVM to 6GB (double) did not help either.  We were seeing higher
CPU and higher heap usage.

We later changed cache settings so as to reduce their sizes, increased the
JVM to 8GB and we see an improvement.  But over time, we do see that the
Heap utilization slowly climbs as the 200QPS test is allowed to run, and
sometimes leads to max heap being exceeded from the JConsole.  So we see the
jagged edge waveform which keeps climbing (GC cycles don't completely
collect memory over time).  Our test has a short capture from real traffic
and we are replaying that via solrmeter. 

Thanks.
Regards,
-- Sandeep



--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-2-1-higher-memory-footprint-vs-Solr-3-5-tp4067879p4068150.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr 4.2.1 higher memory footprint vs Solr 3.5

Posted by Shawn Heisey <so...@elyograg.org>.
On 6/4/2013 6:06 AM, Erick Erickson wrote:
> About shipping data back to the client. Well, you
> say the documents are very large Solr 4.2
> automatically compresses stored data, you might
> be spending a lot if time decompressing the data.
> You could try turning that option off. Other than

Erick,

I'm not aware of any option in Solr to turn compression off.  I've been
told that it would require creating a special codec that doesn't do
compression and somehow referencing that.

On a modern system with documents that aren't huge, there is typically
not a lot of difference in performance when Solr uses compression, but
there have been cases with older hardware or with atypical documents
where the compression has resulted in a major performance drop.

I think it would be a really good idea to have a configuration knob to
turn compression off.  If that would be difficult to support when
upgrading, we should include alternate codecs, with the understanding
that some version upgrades are likely to require wiping out the index
and building it again.

Thanks,
Shawn


Re: Solr 4.2.1 higher memory footprint vs Solr 3.5

Posted by Erick Erickson <er...@gmail.com>.
While the schema may be the same, did you also
closely examine solrconfig.xml, particularly cache
settings?

Memory _should_ have gone down significantly,
especially if you're dealing with much textual data
or string types. So I'm guessing something
innocent in your configs is showing this difference.

Try taking a heap dump of each, you should see
something significantly different....

About shipping data back to the client. Well, you
say the documents are very large Solr 4.2
automatically compresses stored data, you might
be spending a lot if time decompressing the data.
You could try turning that option off. Other than
that, has your environment changed in any way
especially your connection to disk? Check the CPU
usage on the server, if you see lots of CPU then
I'd look at compression first. If you see lots of I/O wait,
then disk access is where I'd start looking.

Best
Erick

On Mon, Jun 3, 2013 at 12:34 PM, SandeepM <sk...@hotmail.com> wrote:
> Hi,
>
> Using the same schema for both Solr 3.5 and Solr 4.2.1 and posting the same
> data to both these server,  and the memory requirements seem to have gone up
> sharply during request handling.
> . Requests come in at around 200QPS.
> . Document sizes are very large but that did not seem to be a problem with
> 3.5 (Lots of multivalued fields with large array lengths.)
> Could you help me understand what change in SOLR 4.2.1 would attribute to
> this higher memory requirement?
>
> Also, in a different test, I ran a query to just get a list of all unique
> ID's via a single query and no load and I see it complete in <500ms however
> the time it takes to ship the data back to the client seems to be very
> large.  Any idea what could be causing this behavior?
>
> Would appreciate any help.
>
> Regards,
> -- Sandeep
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-2-1-higher-memory-footprint-vs-Solr-3-5-tp4067879.html
> Sent from the Solr - User mailing list archive at Nabble.com.