You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Marc Sturlese <ma...@gmail.com> on 2011/06/15 13:25:33 UTC
problem with the new IndexSearcher when snpainstaller (and commit
script) happen
Hey there,
I've noticed a very odd behaviour with the snapinstaller and commit (using
collectionDistribution scripts). The first time I install a new index
everything works fine. But when installing a new one, I can't see the new
documents. Checking the status page of the core tells me that the index
version has changed but numDocs and maxDocs are the same. I have a simple
script that get the version form an index reader and this confirms me that
that's not true. numDocs and maxDocs are different in both indexs.
The index I'm trying to install is a whole new index, generated with
mergefactor = 2 and optimized with no compound file.
I've tried manually to mv index to index.old and the snapshot.xxxxx to index
(while tomcat is up) and manually execute:
curl http://localhost:8080/trovit_solr/coreA/update?commit=true -H
"Content-Type: text/xml"
But the same is happening.
Checking the logs I can see that apparently everything is fine. New searcher
is registered and warming is properly done to it.
I would think that the problem is with some reference opening the index
searcher. But the fact that the indexVersion changes but the numDocs and
maxDocs dont' makes me understand nothing.
If I reload the core, numDocs and maxDocs changes and everything is fine.
Any idea what could be happening here?
Thanks in advance.
--
View this message in context: http://lucene.472066.n3.nabble.com/problem-with-the-new-IndexSearcher-when-snpainstaller-and-commit-script-happen-tp3066902p3066902.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: problem with the new IndexSearcher when snpainstaller (and
commit script) happen
Posted by pravesh <su...@yahoo.com>.
Try to look for snapshot.current file in the logs folder in ur SOLR-home dist
in your slave server, if this shows the older snapshot.
I also faced the similar issue(but with SOLR 1.2.1), using the
collection-distribution scripts.
The way i resolved it was:
1. Stopped the index replication script(s) & index updation scripts.
2. cleaning the slave(s) status directory from the master (keep the status
directory & only delete its contents)
3. removed the snapshot.current file from salve's [SOLR-Home]/log folder
4. restart the snapshooter on master and snappuller on slave(s)
Hope this helps
Thanx
Pravesh
--
View this message in context: http://lucene.472066.n3.nabble.com/problem-with-the-new-IndexSearcher-when-snpainstaller-and-commit-script-happen-tp3066902p3068903.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: problem with the new IndexSearcher when snpainstaller (and
commit script) happen [SOLVED]
Posted by Marc Sturlese <ma...@gmail.com>.
I've found the problem in case someone is interested.
It's because of the indexReader.reopen(). If it is enabled, when opening a
new searcher due to the commit, this code is executed (in
SolrCore.getSearcher(boolean forceNew, boolean returnSearcher, final
Future[] waitSearcher)):
...
if (newestSearcher != null && solrConfig.reopenReaders
&& indexDirFile.equals(newIndexDirFile)) {
IndexReader currentReader = newestSearcher.get().getReader();
IndexReader newReader = currentReader.reopen();
if (newReader == currentReader) {
currentReader.incRef();
}
tmp = new SolrIndexSearcher(this, schema, "main", newReader, true,
true);
} else {
IndexReader reader =
getIndexReaderFactory().newReader(getDirectoryFactory().open(newIndexDir),
true);
tmp = new SolrIndexSearcher(this, schema, "main", reader, true,
true);
}
...
If the names of the segments haven't changed, IndexReader.reopen thinks that
they haven't actually changed (but they have in my case as index files have
same name but have different docs at the same time) so instead of opening
new reader for the segments, it gives back the same one and changes can't be
seen by the new IndexSearcher.
Knowing that performance gets worse, disabling reopen in solrconfig.xml
solves the problem (and is still better performance that reloading the whole
core).
Does someone knows that this still happen on Lucene 3.2?
--
View this message in context: http://lucene.472066.n3.nabble.com/problem-with-the-new-IndexSearcher-when-snpainstaller-and-commit-script-happen-tp3066902p3068956.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: problem with the new IndexSearcher when snpainstaller (and
commit script) happen
Posted by Marc Sturlese <ma...@gmail.com>.
I have some more info!
I've build another index bigger than the others so names of the files are
not the same. This way, if I move from any of the other index to the bigger
one or vicevera it works (I can see the cahnges in the version, numDocs and
maxDocs)! So, I thing it is related to the name of the files.
Maybe the server gets confused with the pointers of the older index files or
something like that?
The bigger index looks like:
./index.big:
total 4181088
drwxr-xr-x 12 marc admin 408 15 Jun 16:46 .
drwxr-xr-x 11 marc admin 374 15 Jun 16:48 ..
-rw-r--r-- 1 marc admin 1666038160 15 Jun 16:43 _4.fdt
-rw-r--r-- 1 marc admin 9178780 15 Jun 16:43 _4.fdx
-rw-r--r-- 1 marc admin 477 15 Jun 16:43 _4.fnm
-rw-r--r-- 1 marc admin 232687972 15 Jun 16:44 _4.frq
-rw-r--r-- 1 marc admin 4589392 15 Jun 16:44 _4.nrm
-rw-r--r-- 1 marc admin 161931683 15 Jun 16:45 _4.prx
-rw-r--r-- 1 marc admin 824985 15 Jun 16:45 _4.tii
-rw-r--r-- 1 marc admin 65438631 15 Jun 16:45 _4.tis
-rw-r--r-- 1 marc admin 20 15 Jun 16:45 segments.gen
-rw-r--r-- 1 marc admin 298 15 Jun 16:45 segments_2
--
View this message in context: http://lucene.472066.n3.nabble.com/problem-with-the-new-IndexSearcher-when-snpainstaller-and-commit-script-happen-tp3066902p3067657.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: problem with the new IndexSearcher when snpainstaller (and
commit script) happen
Posted by Marc Sturlese <ma...@gmail.com>.
I don't know if this could have something to do with the problem but some of
the files of the indexes have same size and name (in all the index but not
in the empty one).
I have also realized that when moving back to the empty index and
committing, numDocs and maxDocs change. Once I'm with the empty index, if I
move to another one it works too. The problem happens when moving to a none
empty index to another none empty index.
That's because I think that the name and size of some files could have
something to do with the problem:
./index.1:
total 702024
drwxr-sr-x 12 marc admin 408 15 Jun 11:01 .
drwxr-xr-x 10 marc admin 340 15 Jun 16:04 ..
-rw-r--r-- 1 marc admin 269347737 15 Jun 10:57 _3.fdt
-rw-r--r-- 1 marc admin 2067804 15 Jun 10:57 _3.fdx
-rw-r--r-- 1 marc admin 463 15 Jun 10:57 _3.fnm
-rw-r--r-- 1 marc admin 40372030 15 Jun 10:59 _3.frq
-rw-r--r-- 1 marc admin 1033904 15 Jun 10:59 _3.nrm
-rw-r--r-- 1 marc admin 27021337 15 Jun 11:00 _3.prx
-rw-r--r-- 1 marc admin 234891 15 Jun 11:00 _3.tii
-rw-r--r-- 1 marc admin 19330416 15 Jun 11:01 _3.tis
-rw-r--r-- 1 marc admin 20 15 Jun 11:01 segments.gen
-rw-r--r-- 1 marc admin 298 15 Jun 11:01 segments_2
./index.2:
total 701296
drwxr-sr-x 12 marc admin 408 15 Jun 11:11 .
drwxr-xr-x 10 marc admin 340 15 Jun 16:04 ..
-rw-r--r-- 1 marc admin 269044254 15 Jun 11:09 _3.fdt
-rw-r--r-- 1 marc admin 2068116 15 Jun 11:09 _3.fdx
-rw-r--r-- 1 marc admin 463 15 Jun 11:09 _3.fnm
-rw-r--r-- 1 marc admin 40320465 15 Jun 11:10 _3.frq
-rw-r--r-- 1 marc admin 1034060 15 Jun 11:10 _3.nrm
-rw-r--r-- 1 marc admin 26967519 15 Jun 11:11 _3.prx
-rw-r--r-- 1 marc admin 235895 15 Jun 11:11 _3.tii
-rw-r--r-- 1 marc admin 19372446 15 Jun 11:11 _3.tis
-rw-r--r-- 1 marc admin 20 15 Jun 11:11 segments.gen
-rw-r--r-- 1 marc admin 298 15 Jun 11:11 segments_2
./index.empty:
total 16
drwxr-xr-x 4 marc admin 136 15 Jun 10:45 .
drwxr-xr-x 10 marc admin 340 15 Jun 16:04 ..
-rw-r--r-- 1 marc admin 20 15 Jun 10:45 segments.gen
-rw-r--r-- 1 marc admin 32 15 Jun 10:45 segments_1
--
View this message in context: http://lucene.472066.n3.nabble.com/problem-with-the-new-IndexSearcher-when-snpainstaller-and-commit-script-happen-tp3066902p3067466.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: problem with the new IndexSearcher when snpainstaller (and
commit script) happen
Posted by Marc Sturlese <ma...@gmail.com>.
Test are done on Solr 1.4
The simplest way to reproduce my problem is having 2 indexes and a Solr box
with just one core. Both index must have been created with the same schema.
1- Remove the index dir of the core and start the server (core is up with an
empty index)
2- check status page of the core (version,numDocs,maxDocs version should be
X and numDocs,maxDocs zero)
3- mv index to index.old
4- mv your folderA (wich contains an index) to index
5- execute curl http://localhost:8080/solr/coreA/update?commit=true -H
"Content-Type: text/xml"
* Here the log shows me that commit has been executed and new IndexSearcher
has been registered and proper warming has been done.
6- Check the core status page (here all has changed:
version,numDocs,maxDocs)
If now I repeat steps 3,4,5 (in this case using folderB with another index),
when I do step 6, indexVersion has changed but numDocs and maxDocs stay the
same, which I can't understand in any way. (opening the index with my script
shows me that are not).
I've ended up doing this test after noticing the problem before with the
snapinstaller and commit.
--
View this message in context: http://lucene.472066.n3.nabble.com/problem-with-the-new-IndexSearcher-when-snpainstaller-and-commit-script-happen-tp3066902p3067042.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: problem with the new IndexSearcher when snpainstaller (and commit
script) happen
Posted by Yonik Seeley <yo...@lucidimagination.com>.
What version of Solr is this?
Can you show steps to reproduce w/ the example server and data?
-Yonik
http://www.lucidimagination.com
On Wed, Jun 15, 2011 at 7:25 AM, Marc Sturlese <ma...@gmail.com> wrote:
> Hey there,
> I've noticed a very odd behaviour with the snapinstaller and commit (using
> collectionDistribution scripts). The first time I install a new index
> everything works fine. But when installing a new one, I can't see the new
> documents. Checking the status page of the core tells me that the index
> version has changed but numDocs and maxDocs are the same. I have a simple
> script that get the version form an index reader and this confirms me that
> that's not true. numDocs and maxDocs are different in both indexs.
> The index I'm trying to install is a whole new index, generated with
> mergefactor = 2 and optimized with no compound file.
>
> I've tried manually to mv index to index.old and the snapshot.xxxxx to index
> (while tomcat is up) and manually execute:
> curl http://localhost:8080/trovit_solr/coreA/update?commit=true -H
> "Content-Type: text/xml"
> But the same is happening.
> Checking the logs I can see that apparently everything is fine. New searcher
> is registered and warming is properly done to it.
>
> I would think that the problem is with some reference opening the index
> searcher. But the fact that the indexVersion changes but the numDocs and
> maxDocs dont' makes me understand nothing.
>
> If I reload the core, numDocs and maxDocs changes and everything is fine.
>
> Any idea what could be happening here?
> Thanks in advance.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/problem-with-the-new-IndexSearcher-when-snpainstaller-and-commit-script-happen-tp3066902p3066902.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>