You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by talhanather <ta...@gmail.com> on 2019/01/16 12:57:29 UTC
Re: Starting optimize... Reading and rewriting the entire index!
Use with care
Hi Erick,
PFB the solr-config.xml, Its not having optimization tag to true.
Then how optimization is continuously occurring for me. ?
<requestHandler
class="org.apache.solr.handler.dataimport.DataImportHandler"
name="/dataimport">
<lst name="defaults">
<str name="update.chain">uuid</str>
<str name="config">db-data-config.xml</str>
</lst>
</requestHandler>
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Starting optimize... Reading and rewriting the entire index! Use
with care
Posted by Walter Underwood <wu...@wunderwood.org>.
Agreed, make both of those changes.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Jan 16, 2019, at 7:21 AM, Erick Erickson <er...@gmail.com> wrote:
>
> +1 to defaulting it to false. Maybe call it forceMerge to be more in
> line with Lucene for all the reasons that we've discussed elsewhere?
>
> Talhanather:
>
> Frankly, I'd just stop optimizing. It's mis-named and there are a
> series of very good reasons to avoid it _unless_
> 1> you can _measure_ a significant improvement after the op
> 2> you can afford the time etc. to do it every time.
>
> See: https://lucidworks.com/2017/10/13/segment-merging-deleted-documents-optimize-may-bad/
> It's not as bad, but still expensive in Solr 7.5 and later:
> https://lucidworks.com/2018/06/20/solr-and-optimizing-your-index-take-ii/
>
> If you see that message in your logs, then you're optimizing and need
> do nothing. If you absolutely
> feel you must, use .../update?optimize=true
>
> Best,
> Erick
>
> On Wed, Jan 16, 2019 at 6:38 AM Jan Høydahl <ja...@cominvent.com> wrote:
>>
>> Should we consider to default optimize to false in the DIH UI?
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>>> 16. jan. 2019 kl. 14:23 skrev Jeremy Smith <ja...@cornell.edu>:
>>>
>>> How are you calling the dataimport? As I understand it, optimize defaults to true, so unless you explicitly set it to false, the optimize will occur after the import.
>>>
>>>
>>> ________________________________
>>> From: talhanather <ta...@gmail.com>
>>> Sent: Wednesday, January 16, 2019 7:57:29 AM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: Starting optimize... Reading and rewriting the entire index! Use with care
>>>
>>> Hi Erick,
>>>
>>> PFB the solr-config.xml, Its not having optimization tag to true.
>>> Then how optimization is continuously occurring for me. ?
>>>
>>>
>>>
>>>
>>> <requestHandler
>>> class="org.apache.solr.handler.dataimport.DataImportHandler"
>>> name="/dataimport">
>>> <lst name="defaults">
>>> <str name="update.chain">uuid</str>
>>> <str name="config">db-data-config.xml</str>
>>> </lst>
>>> </requestHandler>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>
Re: Starting optimize... Reading and rewriting the entire index! Use
with care
Posted by Erick Erickson <er...@gmail.com>.
+1 to defaulting it to false. Maybe call it forceMerge to be more in
line with Lucene for all the reasons that we've discussed elsewhere?
Talhanather:
Frankly, I'd just stop optimizing. It's mis-named and there are a
series of very good reasons to avoid it _unless_
1> you can _measure_ a significant improvement after the op
2> you can afford the time etc. to do it every time.
See: https://lucidworks.com/2017/10/13/segment-merging-deleted-documents-optimize-may-bad/
It's not as bad, but still expensive in Solr 7.5 and later:
https://lucidworks.com/2018/06/20/solr-and-optimizing-your-index-take-ii/
If you see that message in your logs, then you're optimizing and need
do nothing. If you absolutely
feel you must, use .../update?optimize=true
Best,
Erick
On Wed, Jan 16, 2019 at 6:38 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
> Should we consider to default optimize to false in the DIH UI?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 16. jan. 2019 kl. 14:23 skrev Jeremy Smith <ja...@cornell.edu>:
> >
> > How are you calling the dataimport? As I understand it, optimize defaults to true, so unless you explicitly set it to false, the optimize will occur after the import.
> >
> >
> > ________________________________
> > From: talhanather <ta...@gmail.com>
> > Sent: Wednesday, January 16, 2019 7:57:29 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Starting optimize... Reading and rewriting the entire index! Use with care
> >
> > Hi Erick,
> >
> > PFB the solr-config.xml, Its not having optimization tag to true.
> > Then how optimization is continuously occurring for me. ?
> >
> >
> >
> >
> > <requestHandler
> > class="org.apache.solr.handler.dataimport.DataImportHandler"
> > name="/dataimport">
> > <lst name="defaults">
> > <str name="update.chain">uuid</str>
> > <str name="config">db-data-config.xml</str>
> > </lst>
> > </requestHandler>
> >
> >
> >
> >
> > --
> > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Re: Starting optimize... Reading and rewriting the entire index! Use
with care
Posted by Jan Høydahl <ja...@cominvent.com>.
Should we consider to default optimize to false in the DIH UI?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 16. jan. 2019 kl. 14:23 skrev Jeremy Smith <ja...@cornell.edu>:
>
> How are you calling the dataimport? As I understand it, optimize defaults to true, so unless you explicitly set it to false, the optimize will occur after the import.
>
>
> ________________________________
> From: talhanather <ta...@gmail.com>
> Sent: Wednesday, January 16, 2019 7:57:29 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Starting optimize... Reading and rewriting the entire index! Use with care
>
> Hi Erick,
>
> PFB the solr-config.xml, Its not having optimization tag to true.
> Then how optimization is continuously occurring for me. ?
>
>
>
>
> <requestHandler
> class="org.apache.solr.handler.dataimport.DataImportHandler"
> name="/dataimport">
> <lst name="defaults">
> <str name="update.chain">uuid</str>
> <str name="config">db-data-config.xml</str>
> </lst>
> </requestHandler>
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Starting optimize... Reading and rewriting the entire index! Use
with care
Posted by Jeremy Smith <ja...@cornell.edu>.
How are you calling the dataimport? As I understand it, optimize defaults to true, so unless you explicitly set it to false, the optimize will occur after the import.
________________________________
From: talhanather <ta...@gmail.com>
Sent: Wednesday, January 16, 2019 7:57:29 AM
To: solr-user@lucene.apache.org
Subject: Re: Starting optimize... Reading and rewriting the entire index! Use with care
Hi Erick,
PFB the solr-config.xml, Its not having optimization tag to true.
Then how optimization is continuously occurring for me. ?
<requestHandler
class="org.apache.solr.handler.dataimport.DataImportHandler"
name="/dataimport">
<lst name="defaults">
<str name="update.chain">uuid</str>
<str name="config">db-data-config.xml</str>
</lst>
</requestHandler>
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html