Re: Frequent deletions

[Michael McCandless <lu...@mikemccandless.com>]

Also see this G+ post I wrote up recently showing how %tg deletions
changes over time for an "every add also deletes a previous document"
stress test: https://plus.google.com/112759599082866346694/posts/MJVueTznYnD

Mike McCandless

http://blog.mikemccandless.com


On Wed, Dec 31, 2014 at 12:21 PM, Erick Erickson
<er...@gmail.com> wrote:
> It's usually not necessary to optimize, as more indexing happens you
> should see background merges happen that'll reclaim the space, so I
> wouldn't worry about it unless you're seeing actual problems that have
> to be addressed. Here's a great visualization of the process:
>
> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
>
> See especially the third video, "TieredMergePolicy" which is the default.
>
> If you insist, however, try a commit with expungeDeletes=true
>
> and if that isn't enough, try an optimize call
> you can issue a "force merge" (aka optimize)  command from the URL (Or
> cUrl etc) as:
> http://localhost:8983/solr/techproducts/update?optimize=true
>
> But please don't do this unless it's absolutely necessary. You state
> that you have "frequent deletions", but eventually this shoul dall
> happen in the background. Optimize is a fairly expensive operation and
> should be used judiciously.
>
> Best,
> Erick
>
> On Wed, Dec 31, 2014 at 1:32 AM, ig01 <in...@elbitsystems.com> wrote:
>> Hello,
>> We perform frequent deletions from our index, which greatly increases the
>> index size.
>> How can we perform an optimization in order to reduce the size.
>> Please advise,
>> Thanks.
>>
>>
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/Frequent-deletions-tp4176689.html
>> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Frequent deletions

[Alexandre Rafalovitch <ar...@gmail.com>]

Is there a specific list of which data structures are "sparce" and
"non-sparce" for Lucene and Solr (referencing G+ post)? I imagine this
is obvious to low-level hackers, but could actually be nice to
summarize it somewhere for troubleshooting.

Regards,
   Alex.
----
Sign up for my Solr resources newsletter at http://www.solr-start.com/


On 1 January 2015 at 05:22, Michael McCandless
<lu...@mikemccandless.com> wrote:
> Also see this G+ post I wrote up recently showing how %tg deletions
> changes over time for an "every add also deletes a previous document"
> stress test: https://plus.google.com/112759599082866346694/posts/MJVueTznYnD
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Wed, Dec 31, 2014 at 12:21 PM, Erick Erickson
> <er...@gmail.com> wrote:
>> It's usually not necessary to optimize, as more indexing happens you
>> should see background merges happen that'll reclaim the space, so I
>> wouldn't worry about it unless you're seeing actual problems that have
>> to be addressed. Here's a great visualization of the process:
>>
>> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
>>
>> See especially the third video, "TieredMergePolicy" which is the default.
>>
>> If you insist, however, try a commit with expungeDeletes=true
>>
>> and if that isn't enough, try an optimize call
>> you can issue a "force merge" (aka optimize)  command from the URL (Or
>> cUrl etc) as:
>> http://localhost:8983/solr/techproducts/update?optimize=true
>>
>> But please don't do this unless it's absolutely necessary. You state
>> that you have "frequent deletions", but eventually this shoul dall
>> happen in the background. Optimize is a fairly expensive operation and
>> should be used judiciously.
>>
>> Best,
>> Erick
>>
>> On Wed, Dec 31, 2014 at 1:32 AM, ig01 <in...@elbitsystems.com> wrote:
>>> Hello,
>>> We perform frequent deletions from our index, which greatly increases the
>>> index size.
>>> How can we perform an optimization in order to reduce the size.
>>> Please advise,
>>> Thanks.
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://lucene.472066.n3.nabble.com/Frequent-deletions-tp4176689.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.