index copy omits documents

[UnConundrum <wp...@princelaw.com>]

I replicate from a live server to a backup server.  That backup server is
also used for development, so every night by cron, and sometimes manually, I
execute the following script to update a development core on the backup
server:


date
echo "stopping mysql slave"
mysqladmin -u intranet -ppassword stop-slave
echo "stopping solr replication"
curl
http://my_live_server.com:8983/solr/production/admin/replication?command=disablereplication/
echo "copying and uploading mysql tables"
mysqldump -u intranet -ppassword intranet_beta | mysql -u intranet
-ppassword intranet_beta_dev
echo copying production solr index to development
rm -rf /solr/test_site/development/data/*
cp -rf  /solr/test_site/production/data/* /solr/test_site/development/data/
echo "restarting solr replication"
curl
http://my_live_server.com:8983/solr/production/admin/replication?command=enablereplication/
echo "restarting mysql replication/slave"
mysqladmin -u intranet -ppassword start-slave
echo "finished"
date


I noticed last night that many documents are being omitted.  On the
production core, numDocs is 2476185, while on the development core numDocs
is 2357785, about 5% less.  The live server reports 2476192 docs (diff is
due to delay on my part).  

Does anyone have any idea why a copy of the directory doesn't result in the
same number of documents?
(FYI, this is affecting my development as queries are missing documents).

Thanks in advance.




--
View this message in context: http://lucene.472066.n3.nabble.com/index-copy-omits-documents-tp4030043.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: index copy omits documents

[UnConundrum <wp...@princelaw.com>]

Well, after hours of fighting with this, I decided to turn on replication
between the production and development cores and use curl commands to
disable replication to the development core and only run replication after
the script updates the development db, doing a sleep 5m, and then disable
replication again.   Seems to be working.



--
View this message in context: http://lucene.472066.n3.nabble.com/index-copy-omits-documents-tp4030043p4030147.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: index copy omits documents

[UnConundrum <wp...@princelaw.com>]

Jack Krupansky-2 wrote
> Do you have any soft commits ble.com.

I don't think so, especially since the Live production core and replicated
production core are only 7 documents apart. I haven't used the copied
development core at all, nor do I ever use the replicated production core
since it serves as a backup to the live production core and I want to keep
it pristine.



--
View this message in context: http://lucene.472066.n3.nabble.com/index-copy-omits-documents-tp4030043p4030074.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: index copy omits documents

[Jack Krupansky <ja...@basetechnology.com>]

Do you have any soft commits (or commitWithin that is a soft commit) that 
show up in queries but haven't yet been committed to disk? You have to do a 
hard commit to flush those to disk.

-- Jack Krupansky

-----Original Message----- 
From: UnConundrum
Sent: Wednesday, January 02, 2013 8:42 AM
To: solr-user@lucene.apache.org
Subject: index copy omits documents

I replicate from a live server to a backup server.  That backup server is
also used for development, so every night by cron, and sometimes manually, I
execute the following script to update a development core on the backup
server:


date
echo "stopping mysql slave"
mysqladmin -u intranet -ppassword stop-slave
echo "stopping solr replication"
curl
http://my_live_server.com:8983/solr/production/admin/replication?command=disablereplication/
echo "copying and uploading mysql tables"
mysqldump -u intranet -ppassword intranet_beta | mysql -u intranet
-ppassword intranet_beta_dev
echo copying production solr index to development
rm -rf /solr/test_site/development/data/*
cp -rf  /solr/test_site/production/data/* /solr/test_site/development/data/
echo "restarting solr replication"
curl
http://my_live_server.com:8983/solr/production/admin/replication?command=enablereplication/
echo "restarting mysql replication/slave"
mysqladmin -u intranet -ppassword start-slave
echo "finished"
date


I noticed last night that many documents are being omitted.  On the
production core, numDocs is 2476185, while on the development core numDocs
is 2357785, about 5% less.  The live server reports 2476192 docs (diff is
due to delay on my part).

Does anyone have any idea why a copy of the directory doesn't result in the
same number of documents?
(FYI, this is affecting my development as queries are missing documents).

Thanks in advance.




--
View this message in context: 
http://lucene.472066.n3.nabble.com/index-copy-omits-documents-tp4030043.html
Sent from the Solr - User mailing list archive at Nabble.com.