You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by a sd <li...@gmail.com> on 2012/04/05 05:23:20 UTC
It cost some many memory with solrj 3.5 & how to decrease it?
hi,all.
I have write a program which send data to solr using the "update"
request handler, when i adopted server & client library ( namely solrj )
with version 4.0 or 3.2, jvm`s heap size was up to 1.0 G about, but ,when i
transfer the all of them to solr 3.5 ( both server and client libs), the
size of heap was top to 3.0G ! There are the same server configuration and
the identical program. What`s wrong with the new version of solrj 3.5 , i
had looked the source code, there is no difference between solrj 3.2 and
solrj 3.5 where my program may invoke. How can i do to decrease the memory
cost by solrj 3.5?
Any advice will be appreciated!
murphy
Re: It cost some many memory with solrj 3.5 & how to decrease it?
Posted by a sd <li...@gmail.com>.
Study the update examination more deeply,i logged all "elapsetime" value of
Updateresponse, the result list following:
It seems that it spent almost 20 ms on adding/updating one document in
general, thus, i called which spend less than 20ms on adding one docs as
normal log,and the others were "abnormal" logs.
i can`t get a correct suit of solr 1.4, i use solr3.2 which has same
performance as solr 1.4 during the test.
solr3.5 vs solr 3.2
solr3.5
sum of docs:31998
sum of elapsetime:1218344 ms
average: 38.0744 ms /doc
sum of normal docs:28409
sum of normal elapsetime:442258
average=15.5675 ms/doc
normal percentage:28409/31998 = 88.78%
abnormal docs: 3590
solr 3.2
sum of docs:31998
sum of elapsetime:852935 ms
average:26.6559 ms /doc
sum of normal docs:28416
sum of normal elapsetime:443045
average=15.5914 ms/doc
normal percentage:28409/31998 = 88.80%
abnormal docs: 3160
What can be analyzed from them?
B.R.
murphy
On Fri, Apr 6, 2012 at 10:28 AM, a sd <liurx.cn@gma
il.com> wrote:
> hi,Erick.
> thanks at first.
> I had watched the status of JVM at runtime helped by "jconsole" and
> "jmap".
> 1,When the "Xmx" was not assigned, then, the "Old Gen" area was full whose
> size was up to 1.5Gb and whose major content are instances of "String" ,
> when the whole size of heap was up to the maximum ( about 2GB), the JVM run
> gc() ,which wasted the CPU time,then, the performance was degraded sharply,
> which was from 100,000 docs per minute to 10,000 docs per minute, as a
> examination, i assigned "Xmx=1024m" purposely, the amount was down to 1000
> docs per minute.
> 2,When assigned "Xmx=4096m", i found that the "Old Gen" was up to 2.1 GB
> and the all size of JVM was up to 3GB, but, the performance with 100,000
> docs per minute can attained.
> During all of the test above, i only adjust the setting of client, which
> connect to the identical solr server and i empty the "data" directory of
> solr home before every test.
> By the way, i know the client code was very ugly occupied so many heap
> too, but, i wan`t permitted to promote them before i obtain a benchrank
> using solrj 3.5 as much as which the old version did using solrj 1.4.
> B.R
> murphy
>
>
> On Fri, Apr 6, 2012 at 5:54 AM, Erick Erickson <er...@gmail.com>wrote:
>
>> "What's memory"? Really, how are you measuring it?
>>
>> If it's virtual, you don't need to worry about it. Is this
>> causing you a real problem or are you just nervous about
>> the difference?
>>
>> Best
>> Erick
>>
>> On Wed, Apr 4, 2012 at 11:23 PM, a sd <li...@gmail.com> wrote:
>> > hi,all.
>> > I have write a program which send data to solr using the "update"
>> > request handler, when i adopted server & client library ( namely solrj )
>> > with version 4.0 or 3.2, jvm`s heap size was up to 1.0 G about, but
>> ,when i
>> > transfer the all of them to solr 3.5 ( both server and client libs), the
>> > size of heap was top to 3.0G ! There are the same server configuration
>> and
>> > the identical program. What`s wrong with the new version of solrj 3.5 ,
>> i
>> > had looked the source code, there is no difference between solrj 3.2 and
>> > solrj 3.5 where my program may invoke. How can i do to decrease the
>> memory
>> > cost by solrj 3.5?
>> > Any advice will be appreciated!
>> > murphy
>>
>
>
Re: It cost some many memory with solrj 3.5 & how to decrease it?
Posted by a sd <li...@gmail.com>.
hi,Erick.
thanks at first.
I had watched the status of JVM at runtime helped by "jconsole" and "jmap".
1,When the "Xmx" was not assigned, then, the "Old Gen" area was full whose
size was up to 1.5Gb and whose major content are instances of "String" ,
when the whole size of heap was up to the maximum ( about 2GB), the JVM run
gc() ,which wasted the CPU time,then, the performance was degraded sharply,
which was from 100,000 docs per minute to 10,000 docs per minute, as a
examination, i assigned "Xmx=1024m" purposely, the amount was down to 1000
docs per minute.
2,When assigned "Xmx=4096m", i found that the "Old Gen" was up to 2.1 GB
and the all size of JVM was up to 3GB, but, the performance with 100,000
docs per minute can attained.
During all of the test above, i only adjust the setting of client, which
connect to the identical solr server and i empty the "data" directory of
solr home before every test.
By the way, i know the client code was very ugly occupied so many heap too,
but, i wan`t permitted to promote them before i obtain a benchrank using
solrj 3.5 as much as which the old version did using solrj 1.4.
B.R
murphy
On Fri, Apr 6, 2012 at 5:54 AM, Erick Erickson <er...@gmail.com>wrote:
> "What's memory"? Really, how are you measuring it?
>
> If it's virtual, you don't need to worry about it. Is this
> causing you a real problem or are you just nervous about
> the difference?
>
> Best
> Erick
>
> On Wed, Apr 4, 2012 at 11:23 PM, a sd <li...@gmail.com> wrote:
> > hi,all.
> > I have write a program which send data to solr using the "update"
> > request handler, when i adopted server & client library ( namely solrj )
> > with version 4.0 or 3.2, jvm`s heap size was up to 1.0 G about, but
> ,when i
> > transfer the all of them to solr 3.5 ( both server and client libs), the
> > size of heap was top to 3.0G ! There are the same server configuration
> and
> > the identical program. What`s wrong with the new version of solrj 3.5 , i
> > had looked the source code, there is no difference between solrj 3.2 and
> > solrj 3.5 where my program may invoke. How can i do to decrease the
> memory
> > cost by solrj 3.5?
> > Any advice will be appreciated!
> > murphy
>
Re: It cost some many memory with solrj 3.5 & how to decrease it?
Posted by Erick Erickson <er...@gmail.com>.
"What's memory"? Really, how are you measuring it?
If it's virtual, you don't need to worry about it. Is this
causing you a real problem or are you just nervous about
the difference?
Best
Erick
On Wed, Apr 4, 2012 at 11:23 PM, a sd <li...@gmail.com> wrote:
> hi,all.
> I have write a program which send data to solr using the "update"
> request handler, when i adopted server & client library ( namely solrj )
> with version 4.0 or 3.2, jvm`s heap size was up to 1.0 G about, but ,when i
> transfer the all of them to solr 3.5 ( both server and client libs), the
> size of heap was top to 3.0G ! There are the same server configuration and
> the identical program. What`s wrong with the new version of solrj 3.5 , i
> had looked the source code, there is no difference between solrj 3.2 and
> solrj 3.5 where my program may invoke. How can i do to decrease the memory
> cost by solrj 3.5?
> Any advice will be appreciated!
> murphy