You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rohit <ro...@in-rev.com> on 2012/04/12 06:45:48 UTC
Solr 3.5 takes very long to commit gradually
We recently migrated from solr3.1 to solr3.5, we have one master and one
slave configured. The master has two cores,
1) Core1 - 44555972 documents
2) Core2 - 29419244 documents
We commit every 5000 documents, but lately the commit time gradually
increase and solr is taking as very long 15 minutes plus in some cases. What
could have caused this, I have checked the logs and the only warning i can
see is,
"WARNING: Use of deprecated update request parameter update.processor
detected. Please use the new parameter update.chain instead, as support for
update.processor will be removed in a later version."
Memory details:
export JAVA_OPTS="$JAVA_OPTS -Xms6g -Xmx36g -XX:MaxPermSize=5g"
Solr Config:
<useCompoundFile>false</useCompoundFile>
<mergeFactor>10</mergeFactor>
<ramBufferSizeMB>32</ramBufferSizeMB>
<!-- <maxBufferedDocs>1000</maxBufferedDocs> -->
<maxFieldLength>10000</maxFieldLength>
<writeLockTimeout>1000</writeLockTimeout>
<commitLockTimeout>10000</commitLockTimeout>
Also noticed, that top command show almost 350GB of Virtual memory usage.
What could be causing this, as everything was running fine a few days back?
Regards,
Rohit
Mobile: +91-9901768202
About Me: <http://about.me/rohitg> http://about.me/rohitg
RE: Solr 3.5 takes very long to commit gradually
Posted by Rohit <ro...@in-rev.com>.
Operating system in linux ubuntu.
No not using spellchecker
Only language detection in my update chain.
Regards,
Rohit
Mobile: +91-9901768202
About Me: http://about.me/rohitg
-----Original Message-----
From: Jan Høydahl [mailto:jan.asf@cominvent.com]
Sent: 12 April 2012 12:50
To: solr-user@lucene.apache.org
Subject: Re: Solr 3.5 takes very long to commit gradually
What operating system?
Are you using spellchecker with buildOnCommit?
Anything special in your Update Chain?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com
On 12. apr. 2012, at 06:45, Rohit wrote:
> We recently migrated from solr3.1 to solr3.5, we have one master and
> one slave configured. The master has two cores,
>
> 1) Core1 - 44555972 documents
>
> 2) Core2 - 29419244 documents
>
> We commit every 5000 documents, but lately the commit time gradually
> increase and solr is taking as very long 15 minutes plus in some
> cases. What could have caused this, I have checked the logs and the
> only warning i can see is,
>
> "WARNING: Use of deprecated update request parameter update.processor
> detected. Please use the new parameter update.chain instead, as
> support for update.processor will be removed in a later version."
>
> Memory details:
>
> export JAVA_OPTS="$JAVA_OPTS -Xms6g -Xmx36g -XX:MaxPermSize=5g"
>
> Solr Config:
>
> <useCompoundFile>false</useCompoundFile>
>
> <mergeFactor>10</mergeFactor>
>
> <ramBufferSizeMB>32</ramBufferSizeMB>
>
> <!-- <maxBufferedDocs>1000</maxBufferedDocs> -->
>
> <maxFieldLength>10000</maxFieldLength>
>
> <writeLockTimeout>1000</writeLockTimeout>
>
> <commitLockTimeout>10000</commitLockTimeout>
>
> Also noticed, that top command show almost 350GB of Virtual memory usage.
>
> What could be causing this, as everything was running fine a few days
back?
>
>
>
>
>
> Regards,
>
> Rohit
>
> Mobile: +91-9901768202
>
> About Me: <http://about.me/rohitg> http://about.me/rohitg
>
>
>
RE: Solr 3.5 takes very long to commit gradually
Posted by Rohit <ro...@in-rev.com>.
Thanks for pointing these out, but I still have one concern, why is the
Virtual Memory running in 300g+?
Regards,
Rohit
-----Original Message-----
From: Tirthankar Chatterjee [mailto:tchatterjee@commvault.com]
Sent: 12 April 2012 13:43
To: solr-user@lucene.apache.org
Subject: Re: Solr 3.5 takes very long to commit gradually
thanks Rohit.. for the information.
On Apr 12, 2012, at 4:08 AM, Rohit wrote:
> Hi Tirthankar,
>
> The average size of documents would be a few Kb's this is mostly
> tweets which are being saved. The two cores are storing different kind
> of data and nothing else.
>
> Regards,
> Rohit
> Mobile: +91-9901768202
> About Me: http://about.me/rohitg
>
> -----Original Message-----
> From: Tirthankar Chatterjee [mailto:tchatterjee@commvault.com]
> Sent: 12 April 2012 13:14
> To: solr-user@lucene.apache.org
> Subject: Re: Solr 3.5 takes very long to commit gradually
>
> Hi Rohit,
> What would be the average size of your documents and also can you
> please share your idea of having 2 cores in the master. I just wanted
> to know the reasoning behind the design.
>
> Thanks in advance
>
> Tirthankar
> On Apr 12, 2012, at 3:19 AM, Jan Høydahl wrote:
>
>> What operating system?
>> Are you using spellchecker with buildOnCommit?
>> Anything special in your Update Chain?
>>
>> --
>> Jan Høydahl, search solution architect Cominvent AS -
>> www.cominvent.com Solr Training - www.solrtraining.com
>>
>> On 12. apr. 2012, at 06:45, Rohit wrote:
>>
>>> We recently migrated from solr3.1 to solr3.5, we have one master and
>>> one slave configured. The master has two cores,
>>>
>>> 1) Core1 - 44555972 documents
>>>
>>> 2) Core2 - 29419244 documents
>>>
>>> We commit every 5000 documents, but lately the commit time gradually
>>> increase and solr is taking as very long 15 minutes plus in some
>>> cases. What could have caused this, I have checked the logs and the
>>> only warning i can see is,
>>>
>>> "WARNING: Use of deprecated update request parameter
>>> update.processor detected. Please use the new parameter update.chain
>>> instead, as support for update.processor will be removed in a later
version."
>>>
>>> Memory details:
>>>
>>> export JAVA_OPTS="$JAVA_OPTS -Xms6g -Xmx36g -XX:MaxPermSize=5g"
>>>
>>> Solr Config:
>>>
>>> <useCompoundFile>false</useCompoundFile>
>>>
>>> <mergeFactor>10</mergeFactor>
>>>
>>> <ramBufferSizeMB>32</ramBufferSizeMB>
>>>
>>> <!-- <maxBufferedDocs>1000</maxBufferedDocs> -->
>>>
>>> <maxFieldLength>10000</maxFieldLength>
>>>
>>> <writeLockTimeout>1000</writeLockTimeout>
>>>
>>> <commitLockTimeout>10000</commitLockTimeout>
>>>
>>> Also noticed, that top command show almost 350GB of Virtual memory
usage.
>>>
>>> What could be causing this, as everything was running fine a few
>>> days
> back?
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>>
>>> Rohit
>>>
>>> Mobile: +91-9901768202
>>>
>>> About Me: <http://about.me/rohitg> http://about.me/rohitg
>>>
>>>
>>>
>>
>
> ******************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material
> for the sole use of the intended recipient. Any unauthorized review,
> use or distribution by others is strictly prohibited. If you have
> received the message in error, please advise the sender by reply email
> and delete the message. Thank you."
> *********************************************************
>
>
******************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or
distribution by others is strictly prohibited. If you have received the
message in error, please advise the sender by reply email and delete the
message. Thank you."
*********************************************************
Re: Solr 3.5 takes very long to commit gradually
Posted by Tirthankar Chatterjee <tc...@commvault.com>.
thanks Rohit.. for the information.
On Apr 12, 2012, at 4:08 AM, Rohit wrote:
> Hi Tirthankar,
>
> The average size of documents would be a few Kb's this is mostly tweets
> which are being saved. The two cores are storing different kind of data and
> nothing else.
>
> Regards,
> Rohit
> Mobile: +91-9901768202
> About Me: http://about.me/rohitg
>
> -----Original Message-----
> From: Tirthankar Chatterjee [mailto:tchatterjee@commvault.com]
> Sent: 12 April 2012 13:14
> To: solr-user@lucene.apache.org
> Subject: Re: Solr 3.5 takes very long to commit gradually
>
> Hi Rohit,
> What would be the average size of your documents and also can you please
> share your idea of having 2 cores in the master. I just wanted to know the
> reasoning behind the design.
>
> Thanks in advance
>
> Tirthankar
> On Apr 12, 2012, at 3:19 AM, Jan Høydahl wrote:
>
>> What operating system?
>> Are you using spellchecker with buildOnCommit?
>> Anything special in your Update Chain?
>>
>> --
>> Jan Høydahl, search solution architect Cominvent AS -
>> www.cominvent.com Solr Training - www.solrtraining.com
>>
>> On 12. apr. 2012, at 06:45, Rohit wrote:
>>
>>> We recently migrated from solr3.1 to solr3.5, we have one master and
>>> one slave configured. The master has two cores,
>>>
>>> 1) Core1 - 44555972 documents
>>>
>>> 2) Core2 - 29419244 documents
>>>
>>> We commit every 5000 documents, but lately the commit time gradually
>>> increase and solr is taking as very long 15 minutes plus in some
>>> cases. What could have caused this, I have checked the logs and the
>>> only warning i can see is,
>>>
>>> "WARNING: Use of deprecated update request parameter update.processor
>>> detected. Please use the new parameter update.chain instead, as
>>> support for update.processor will be removed in a later version."
>>>
>>> Memory details:
>>>
>>> export JAVA_OPTS="$JAVA_OPTS -Xms6g -Xmx36g -XX:MaxPermSize=5g"
>>>
>>> Solr Config:
>>>
>>> <useCompoundFile>false</useCompoundFile>
>>>
>>> <mergeFactor>10</mergeFactor>
>>>
>>> <ramBufferSizeMB>32</ramBufferSizeMB>
>>>
>>> <!-- <maxBufferedDocs>1000</maxBufferedDocs> -->
>>>
>>> <maxFieldLength>10000</maxFieldLength>
>>>
>>> <writeLockTimeout>1000</writeLockTimeout>
>>>
>>> <commitLockTimeout>10000</commitLockTimeout>
>>>
>>> Also noticed, that top command show almost 350GB of Virtual memory usage.
>>>
>>> What could be causing this, as everything was running fine a few days
> back?
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>>
>>> Rohit
>>>
>>> Mobile: +91-9901768202
>>>
>>> About Me: <http://about.me/rohitg> http://about.me/rohitg
>>>
>>>
>>>
>>
>
> ******************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material for the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution by others is strictly prohibited. If you have received the
> message in error, please advise the sender by reply email and delete the
> message. Thank you."
> *********************************************************
>
>
******************Legal Disclaimer***************************
"This communication may contain confidential and privileged
material for the sole use of the intended recipient. Any
unauthorized review, use or distribution by others is strictly
prohibited. If you have received the message in error, please
advise the sender by reply email and delete the message. Thank
you."
*********************************************************
RE: Solr 3.5 takes very long to commit gradually
Posted by Rohit <ro...@in-rev.com>.
Hi Tirthankar,
The average size of documents would be a few Kb's this is mostly tweets
which are being saved. The two cores are storing different kind of data and
nothing else.
Regards,
Rohit
Mobile: +91-9901768202
About Me: http://about.me/rohitg
-----Original Message-----
From: Tirthankar Chatterjee [mailto:tchatterjee@commvault.com]
Sent: 12 April 2012 13:14
To: solr-user@lucene.apache.org
Subject: Re: Solr 3.5 takes very long to commit gradually
Hi Rohit,
What would be the average size of your documents and also can you please
share your idea of having 2 cores in the master. I just wanted to know the
reasoning behind the design.
Thanks in advance
Tirthankar
On Apr 12, 2012, at 3:19 AM, Jan Høydahl wrote:
> What operating system?
> Are you using spellchecker with buildOnCommit?
> Anything special in your Update Chain?
>
> --
> Jan Høydahl, search solution architect Cominvent AS -
> www.cominvent.com Solr Training - www.solrtraining.com
>
> On 12. apr. 2012, at 06:45, Rohit wrote:
>
>> We recently migrated from solr3.1 to solr3.5, we have one master and
>> one slave configured. The master has two cores,
>>
>> 1) Core1 - 44555972 documents
>>
>> 2) Core2 - 29419244 documents
>>
>> We commit every 5000 documents, but lately the commit time gradually
>> increase and solr is taking as very long 15 minutes plus in some
>> cases. What could have caused this, I have checked the logs and the
>> only warning i can see is,
>>
>> "WARNING: Use of deprecated update request parameter update.processor
>> detected. Please use the new parameter update.chain instead, as
>> support for update.processor will be removed in a later version."
>>
>> Memory details:
>>
>> export JAVA_OPTS="$JAVA_OPTS -Xms6g -Xmx36g -XX:MaxPermSize=5g"
>>
>> Solr Config:
>>
>> <useCompoundFile>false</useCompoundFile>
>>
>> <mergeFactor>10</mergeFactor>
>>
>> <ramBufferSizeMB>32</ramBufferSizeMB>
>>
>> <!-- <maxBufferedDocs>1000</maxBufferedDocs> -->
>>
>> <maxFieldLength>10000</maxFieldLength>
>>
>> <writeLockTimeout>1000</writeLockTimeout>
>>
>> <commitLockTimeout>10000</commitLockTimeout>
>>
>> Also noticed, that top command show almost 350GB of Virtual memory usage.
>>
>> What could be causing this, as everything was running fine a few days
back?
>>
>>
>>
>>
>>
>> Regards,
>>
>> Rohit
>>
>> Mobile: +91-9901768202
>>
>> About Me: <http://about.me/rohitg> http://about.me/rohitg
>>
>>
>>
>
******************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or
distribution by others is strictly prohibited. If you have received the
message in error, please advise the sender by reply email and delete the
message. Thank you."
*********************************************************
Re: Solr 3.5 takes very long to commit gradually
Posted by Tirthankar Chatterjee <tc...@commvault.com>.
Hi Rohit,
What would be the average size of your documents and also can you please share your idea of having 2 cores in the master. I just wanted to know the reasoning behind the design.
Thanks in advance
Tirthankar
On Apr 12, 2012, at 3:19 AM, Jan Høydahl wrote:
> What operating system?
> Are you using spellchecker with buildOnCommit?
> Anything special in your Update Chain?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> Solr Training - www.solrtraining.com
>
> On 12. apr. 2012, at 06:45, Rohit wrote:
>
>> We recently migrated from solr3.1 to solr3.5, we have one master and one
>> slave configured. The master has two cores,
>>
>> 1) Core1 - 44555972 documents
>>
>> 2) Core2 - 29419244 documents
>>
>> We commit every 5000 documents, but lately the commit time gradually
>> increase and solr is taking as very long 15 minutes plus in some cases. What
>> could have caused this, I have checked the logs and the only warning i can
>> see is,
>>
>> "WARNING: Use of deprecated update request parameter update.processor
>> detected. Please use the new parameter update.chain instead, as support for
>> update.processor will be removed in a later version."
>>
>> Memory details:
>>
>> export JAVA_OPTS="$JAVA_OPTS -Xms6g -Xmx36g -XX:MaxPermSize=5g"
>>
>> Solr Config:
>>
>> <useCompoundFile>false</useCompoundFile>
>>
>> <mergeFactor>10</mergeFactor>
>>
>> <ramBufferSizeMB>32</ramBufferSizeMB>
>>
>> <!-- <maxBufferedDocs>1000</maxBufferedDocs> -->
>>
>> <maxFieldLength>10000</maxFieldLength>
>>
>> <writeLockTimeout>1000</writeLockTimeout>
>>
>> <commitLockTimeout>10000</commitLockTimeout>
>>
>> Also noticed, that top command show almost 350GB of Virtual memory usage.
>>
>> What could be causing this, as everything was running fine a few days back?
>>
>>
>>
>>
>>
>> Regards,
>>
>> Rohit
>>
>> Mobile: +91-9901768202
>>
>> About Me: <http://about.me/rohitg> http://about.me/rohitg
>>
>>
>>
>
******************Legal Disclaimer***************************
"This communication may contain confidential and privileged
material for the sole use of the intended recipient. Any
unauthorized review, use or distribution by others is strictly
prohibited. If you have received the message in error, please
advise the sender by reply email and delete the message. Thank
you."
*********************************************************
Re: Solr 3.5 takes very long to commit gradually
Posted by Tirthankar Chatterjee <tc...@commvault.com>.
Hi Rohit,
Can you please check the solrconfig.xml in 3.5 and compare it with 3.1 if there are any warming queries specified while opening the searchers after a commit.
Thanks,
Tirthankar
On Apr 12, 2012, at 3:30 AM, Tirthankar Chatterjee wrote:
> Hi Rohit,
> What would be the average size of your documents and also can you please share your idea of having 2 cores in the master. I just wanted to know the reasoning behind the design.
>
> Thanks in advance
>
> Tirthankar
> On Apr 12, 2012, at 3:19 AM, Jan Høydahl wrote:
>
>> What operating system?
>> Are you using spellchecker with buildOnCommit?
>> Anything special in your Update Chain?
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>> Solr Training - www.solrtraining.com
>>
>> On 12. apr. 2012, at 06:45, Rohit wrote:
>>
>>> We recently migrated from solr3.1 to solr3.5, we have one master and one
>>> slave configured. The master has two cores,
>>>
>>> 1) Core1 - 44555972 documents
>>>
>>> 2) Core2 - 29419244 documents
>>>
>>> We commit every 5000 documents, but lately the commit time gradually
>>> increase and solr is taking as very long 15 minutes plus in some cases. What
>>> could have caused this, I have checked the logs and the only warning i can
>>> see is,
>>>
>>> "WARNING: Use of deprecated update request parameter update.processor
>>> detected. Please use the new parameter update.chain instead, as support for
>>> update.processor will be removed in a later version."
>>>
>>> Memory details:
>>>
>>> export JAVA_OPTS="$JAVA_OPTS -Xms6g -Xmx36g -XX:MaxPermSize=5g"
>>>
>>> Solr Config:
>>>
>>> <useCompoundFile>false</useCompoundFile>
>>>
>>> <mergeFactor>10</mergeFactor>
>>>
>>> <ramBufferSizeMB>32</ramBufferSizeMB>
>>>
>>> <!-- <maxBufferedDocs>1000</maxBufferedDocs> -->
>>>
>>> <maxFieldLength>10000</maxFieldLength>
>>>
>>> <writeLockTimeout>1000</writeLockTimeout>
>>>
>>> <commitLockTimeout>10000</commitLockTimeout>
>>>
>>> Also noticed, that top command show almost 350GB of Virtual memory usage.
>>>
>>> What could be causing this, as everything was running fine a few days back?
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>>
>>> Rohit
>>>
>>> Mobile: +91-9901768202
>>>
>>> About Me: <http://about.me/rohitg> http://about.me/rohitg
>>>
>>>
>>>
>>
>
******************Legal Disclaimer***************************
"This communication may contain confidential and privileged
material for the sole use of the intended recipient. Any
unauthorized review, use or distribution by others is strictly
prohibited. If you have received the message in error, please
advise the sender by reply email and delete the message. Thank
you."
*********************************************************
Re: Solr 3.5 takes very long to commit gradually
Posted by Jan Høydahl <ja...@cominvent.com>.
What operating system?
Are you using spellchecker with buildOnCommit?
Anything special in your Update Chain?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com
On 12. apr. 2012, at 06:45, Rohit wrote:
> We recently migrated from solr3.1 to solr3.5, we have one master and one
> slave configured. The master has two cores,
>
> 1) Core1 - 44555972 documents
>
> 2) Core2 - 29419244 documents
>
> We commit every 5000 documents, but lately the commit time gradually
> increase and solr is taking as very long 15 minutes plus in some cases. What
> could have caused this, I have checked the logs and the only warning i can
> see is,
>
> "WARNING: Use of deprecated update request parameter update.processor
> detected. Please use the new parameter update.chain instead, as support for
> update.processor will be removed in a later version."
>
> Memory details:
>
> export JAVA_OPTS="$JAVA_OPTS -Xms6g -Xmx36g -XX:MaxPermSize=5g"
>
> Solr Config:
>
> <useCompoundFile>false</useCompoundFile>
>
> <mergeFactor>10</mergeFactor>
>
> <ramBufferSizeMB>32</ramBufferSizeMB>
>
> <!-- <maxBufferedDocs>1000</maxBufferedDocs> -->
>
> <maxFieldLength>10000</maxFieldLength>
>
> <writeLockTimeout>1000</writeLockTimeout>
>
> <commitLockTimeout>10000</commitLockTimeout>
>
> Also noticed, that top command show almost 350GB of Virtual memory usage.
>
> What could be causing this, as everything was running fine a few days back?
>
>
>
>
>
> Regards,
>
> Rohit
>
> Mobile: +91-9901768202
>
> About Me: <http://about.me/rohitg> http://about.me/rohitg
>
>
>