Optimize not working, Solr version 9.1.0-SNAPSHOT

[Shawn Heisey <ap...@elyograg.org>]

While looking into a problem described on the #solr slack channel, I 
tried to have Solr optimize my core. It seems to have completely ignored 
the command.  I am running 9.1.0-SNAPSHOT, compiled from branch_9x.

The user on slack also tried to optimize their index, running version 
8.11.2, and that system behaved the same as mine.

I'm running this Java version, installed as Ubuntu packages:

openjdk version "11.0.16" 2022-07-19
OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu120.04)
OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu120.04, 
mixed mode, sharing)

The user on slack is running:
Oracle Corporation OpenJDK 64-Bit Server VM 11.0.13 11.0.13+8

I have switched to the Shenandoah GC, I am told that the slack user has 
not made any major tweaks to the Solr startup.  They are using the 
docker container, I installed using the included service installer script.

Instead of optimize, they were able to do expungeDeletes, which dropped 
their index from 32GB to 24GB.  My index is only 660MB.  I have no 
deleted docs.

I have been trying to unravel the code to see if maybe optimize has been 
completely disabled (as well as being removed from the admin UI) but 
have not able to figure out where to look.  I have an update processor 
chain defined, but it does not have the processor that ignores commits 
and optimizes:

   <updateRequestProcessorChain name="default" default="true">
     <processor class="solr.TrimFieldUpdateProcessorFactory"/>
     <processor class="solr.LogUpdateProcessorFactory"/>
     <processor class="solr.DistributedUpdateProcessorFactory"/>
     <processor class="solr.RunUpdateProcessorFactory"/>
   </updateRequestProcessorChain>

Below is the last api call I used to try the optimize.  At first I was 
not including the commit option.  Adding it did not help:

curl -v 
"http://localhost:8983/solr/dovecot/update?optimize=true&commit=true"

Does anyone have any ideas why the optimize is not working? Am I doing 
the api call right?

Thanks,
Shawn

Re: Optimize not working, Solr version 9.1.0-SNAPSHOT

[Rahul Goswami <ra...@gmail.com>]

I recently hit this problem on 8.11.1. It was a tiny test index with 2
segments. One of the segments *might* have been from 7.x.  I wanted to
optimize and rewrite the index into a single segment in 8.x . But optimize
didn’t work.

In this case though, before optimize, numDocs=maxDocs. So I thought that
might be the probable cause and probably lucene determines there is no
benefit to optimize. So I induced a delete by doing an atomic update on an
existing document. Then ran optimize with
commit=true&optimize=true&maxSegments=1. Worked this time.

Not sure if this is as designed when numDocs=maxDocs, but was definitely
strange at first.

-Rahul


On Fri, Aug 5, 2022 at 6:46 PM Shawn Heisey <ap...@elyograg.org> wrote:

> While looking into a problem described on the #solr slack channel, I
> tried to have Solr optimize my core. It seems to have completely ignored
> the command.  I am running 9.1.0-SNAPSHOT, compiled from branch_9x.
>
> The user on slack also tried to optimize their index, running version
> 8.11.2, and that system behaved the same as mine.
>
> I'm running this Java version, installed as Ubuntu packages:
>
> openjdk version "11.0.16" 2022-07-19
> OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu120.04)
> OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu120.04,
> mixed mode, sharing)
>
> The user on slack is running:
> Oracle Corporation OpenJDK 64-Bit Server VM 11.0.13 11.0.13+8
>
> I have switched to the Shenandoah GC, I am told that the slack user has
> not made any major tweaks to the Solr startup.  They are using the
> docker container, I installed using the included service installer script.
>
> Instead of optimize, they were able to do expungeDeletes, which dropped
> their index from 32GB to 24GB.  My index is only 660MB.  I have no
> deleted docs.
>
> I have been trying to unravel the code to see if maybe optimize has been
> completely disabled (as well as being removed from the admin UI) but
> have not able to figure out where to look.  I have an update processor
> chain defined, but it does not have the processor that ignores commits
> and optimizes:
>
>    <updateRequestProcessorChain name="default" default="true">
>      <processor class="solr.TrimFieldUpdateProcessorFactory"/>
>      <processor class="solr.LogUpdateProcessorFactory"/>
>      <processor class="solr.DistributedUpdateProcessorFactory"/>
>      <processor class="solr.RunUpdateProcessorFactory"/>
>    </updateRequestProcessorChain>
>
> Below is the last api call I used to try the optimize.  At first I was
> not including the commit option.  Adding it did not help:
>
> curl -v
> "http://localhost:8983/solr/dovecot/update?optimize=true&commit=true"
>
> Does anyone have any ideas why the optimize is not working? Am I doing
> the api call right?
>
> Thanks,
> Shawn
>
>