Issue with Solr Snapshots (missing .nrm file)

[Santhosh Kumar <sa...@autodesk.com>]

We had one issue with our Solr production deployment couple of weeks back.  Following is more info about it.

Server Setup
===============
Platform: Sun Solaris Ultrasparc
JDK: 1.5
Solr: 1.2
Index Size: ~15GB
Topology: One master and two slaves

Problem Statement
===============
Every day we index different contents into the Solr master and run the optimize at the end of it.  Snapshooter is triggered at the end of Optimize and it creates a snapshot of the index.  After couple of hours, Slaves pull the latest snapshot and install it to serve the searches.

Couple of weeks back, Slaves didn't pull the snapshot and when researched we found that one file (.nrm) was missing in the snapshot created by master, as shown below.

# ls -lrth
total 29877000
-rw-r--r--   1 jboss    staff       1.9K Apr 14 11:00 _429o.fnm
-rw-r--r--   1 jboss    staff        28M Apr 14 12:18 _429o.fdx
-rw-r--r--   1 jboss    staff       9.8G Apr 14 12:18 _429o.fdt
-rw-r--r--   1 jboss    staff       501M Apr 14 12:43 _429o.tis
-rw-r--r--   1 jboss    staff       6.5M Apr 14 12:43 _429o.tii
-rw-r--r--   1 jboss    staff       2.4G Apr 14 12:43 _429o.prx
-rw-r--r--   1 jboss    staff       1.2G Apr 14 12:43 _429o.frq
-rw-r--r--   1 jboss    staff         44 Apr 14 12:44 segments_53gt
-rw-r--r--   1 jboss    staff         20 Apr 14 12:44 segments.gen
-rw-r--r--   1 jboss    staff       351M Apr 14 12:44 _429o.nrm <--- This is the missing file in the snapshot!

# cd snapshot.20090414124449/
# ls -lrth
total 29157784
-rw-r--r--   1 jboss    staff         20 Apr 14 12:44 segments.gen
-rw-r--r--   1 jboss    staff       1.9K Apr 14 12:44 _429o.fnm
-rw-r--r--   1 jboss    staff       9.8G Apr 14 13:03 _429o.fdt
-rw-r--r--   1 jboss    staff        28M Apr 14 13:03 _429o.fdx
-rw-r--r--   1 jboss    staff       1.2G Apr 14 13:06 _429o.frq
-rw-r--r--   1 jboss    staff       2.4G Apr 14 13:12 _429o.prx
-rw-r--r--   1 jboss    staff       501M Apr 14 13:13 _429o.tis
-rw-r--r--   1 jboss    staff       6.5M Apr 14 13:13 _429o.tii
-rw-r--r--   1 jboss    staff         44 Apr 14 13:13 segments_53gt
***This snapshot is missing _429o.nrm file!***

Has anybody faced this issue (missing a file, or may be missing .nrm file) might happen?  Any insight is greatly appreciated.

Couple of other questions.


1.       For index of size 15GB, how much of breathing space is required (both memory and diskspace) in master and slave?

2.       Would Slaves pull the snapshots if any of the files are missing in index? (I guess that snapshot-puller will pull but may not be able to install it?)

3.       We are using Solr 1.2 and contemplating to upgrade to 1.3.  What is your experience in this path of upgrade? Is it strongly recommended (based on any critical bugs which were fixed?)

Thanks,
Santhosh.