Replication on startup takes a long time

[Yasoob Haider <ya...@gmail.com>]

Hi

We have setup a master-slave architecture for our Solr instance.

Number of docs: 2 million
Collection size: ~10GB when optimized

We are using Solr 6.2.1.

Autocommit Configuration:

<autoCommit> <maxDocs>40000</maxDocs> <maxTime>900000</maxTime> <
openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit> <maxTime>
${solr.autoSoftCommit.maxTime:3600000}</maxTime> </autoSoftCommit>

I have setup the maxDocs at 40k because we do a heavy weekly indexing, and
I didn't want a lot of commits happening too fast.

Indexing runs smoothly on master. But when I add a new slave pointing to
the master, it takes about 20 minutes for the slave to become queryable.

There are two parts to this latency. First, it takes approximately 13
minutes for the generation of the slave to be same as master. Then it takes
another 7 minutes for the instance to become queryable (it returns 0 hits
in these 7 minutes).

I checked the logs and the collection is downloaded within two minutes.
After that, there is nothing in the logs for next few minutes, even with
LoggingInfoSteam set to 'ALL'.

*Question 1.* What happens after all the files have been downloaded on
slave from master? What is Solr doing internally that the generation sync
up with master takes so long? Whatever it is doing, should it take that
long? (~5 minutes).

After the generation sync up happens, it takes another 7 minutes to start
giving results. I set the autowarm count in all caches to 0, which brought
it down to 3 minutes.

*Question 2. *What is happening here in the 3 minutes? Can this also be
optimized?

And I wanted to ask another unrelated question regarding when a slave
become searchable. I understand that documents on master become searchable
if a hard commit happens with openSearcher set to true, or when a soft
commit happens. But when do documents become searchable on a slave?

*Question 3a. *When do documents become searchable on a slave? As soon as a
segment is copied over from master? Does softcommit make any sense on a
slave, as we are not indexing anything? Does autocommit with opensearcher
true affect slave in any way?

*Question 3b. *Does a softcommit on master affect slave in any way? (I only
have commit and startup options in my replicateAfter field in solrconfig)

Would appreciate any help.

PS: One of my colleague said that the latency may be because our schema.xml
is huge (~500 fields). *Question 4. *Could that be a reason?

Thanks
Yasoob Haider