Optimization of large shard succeeded

[Phillip Farber <pf...@umich.edu>]

I thought I'd summarize a method that solved the problem we were having 
trying to optimize a large shard that was running out of disk space, 
df=100% (400g), du=~380g.  After we ran out of space, if we restarted 
tomcat, segment files disappeared from disk leaving 3 segments.

What worked: we used the <optimize maxSegments=... functionality to 
optimize in maxSegments stages of powers of 2: 16, 8, 4, 2, 1. We did 
not see the merged segment files from previous generations left on disk. 
  The staged optimize was as fast as optimizing once to a single segment 
which was the case which ran out of space.

We were not adding documents to the index. We committed before doing the 
staged optimize. We do not delete documents. We do not use 
replication/distribution/snapshooter. We do not autocommit.

400g LVM volume, 192g/30 segment shard, optimized: 188g

solrconfig:

<useCompoundFile>false</useCompoundFile>
<mergeFactor>10</mergeFactor>
<ramBufferSizeMB>32</ramBufferSizeMB>
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>10000000</maxFieldLength>
<unlockOnStartup>false</unlockOnStartup>
<reopenReaders>true</reopenReaders>
<deletionPolicy class="solr.SolrDeletionPolicy">
    <str name="keepOptimizedOnly">false</str>
    <str name="maxCommitsToKeep">1</str>

schema:

<field name="id" type="string" indexed="true" stored="true" 
required="true"/>
<field name="ocr" type="CommonGramTest" indexed="true" stored="false" 
required="true"/>
<field name="title" type="string" indexed="true" stored="true" 
multiValued="true" required="true"/>
<field name="rights" type="sint" indexed="true" stored="true" 
required="true"/>
<field name="author" type="string" indexed="true" stored="true" 
multiValued="true"/>
<field name="date" type="string" indexed="true" stored="true"/>


Phil
hathitrust.org