You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bernd Fehling <be...@uni-bielefeld.de> on 2013/04/08 10:16:35 UTC
solr 4.2.1 still has problems with index version and index generation
I know there was some effort to fix this but I must report
that solr 4.2.1 has still problems with index version and
index generation numbering in master/slave mode with replication.
Test was:
1. installed solr 4.2.1 on master and build index from scratch
2. installed solr 4.2.1 on slave with empty index
3. replicated master to slave, everything was fine, both in sync
4. deleted index on master with *:* and build index from scratch
5. replicated index from master to slave
RESULT: slave has different (higher) version number and is with generation 1 ahead :-(
Regards
Bernd
Re: solr 4.2.1 still has problems with index version and index generation
Posted by Joel Bernstein <jo...@gmail.com>.
This is the jira issue that addresses this:
https://issues.apache.org/jira/browse/SOLR-4661
I'll try to find some time today and test out the patch and see how things
look.
On Mon, Apr 8, 2013 at 7:18 AM, Tom Gullo <sp...@gmail.com> wrote:
> I'm on 4.1 and I have a similar problem. Except for the version number
> everything else seems to be fine. Is that what other people are seeing?
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/solr-4-2-1-still-has-problems-with-index-version-and-index-generation-tp4054444p4054463.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Joel Bernstein
Professional Services LucidWorks
Re: solr 4.2.1 still has problems with index version and index
generation
Posted by Tom Gullo <sp...@gmail.com>.
I'm on 4.1 and I have a similar problem. Except for the version number
everything else seems to be fine. Is that what other people are seeing?
--
View this message in context: http://lucene.472066.n3.nabble.com/solr-4-2-1-still-has-problems-with-index-version-and-index-generation-tp4054444p4054463.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 4.2.1 still has problems with index version and index
generation
Posted by Chris Hostetter <ho...@fucit.org>.
: And with replication?command=details I also see the correct commit part as
: above, BUT where the hell are the wrong info below the commit array are
: coming from?
Please read the details in the previously mentioned Jira issue...
https://issues.apache.org/jira/browse/SOLR-4661
The indexVersion and generation you are looking at refer to the speciics
of the IndexReader as used by the *seracher* on the master server -- but
in addition to situations like openSearcher=false, there are some
optimizations in place such that Solr/Lucene is smart enough to realize
when an "empty commit" doesn't change the IndexReader it continues to use
the previous commit point...
https://issues.apache.org/jira/browse/SOLR-4661?focusedCommentId=13620195&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13620195
...but from the perspective of the slave, this is still a commit that
needs replicated and loaded.
Hence the current objective of the patch in SOLR-3855: add more details to
the command=details response (as well as the Admin UI) to clearly
distinguish between the gen/ver of the currently replicatable commit and
the gen/ver of the currently open searcher.
All available information suggests that this is purely a problem of
conveying information to users via command=details -- Replication is
behaving as designed using the correct information about hte commit
points.
-Hoss
Re: solr 4.2.1 still has problems with index version and index
generation
Posted by Bernd Fehling <be...@uni-bielefeld.de>.
Looking a bit deeper showed that replication?command=commit reports the
right indexversion, generation and filelist.
<arr name="commits">
<lst>
<long name="indexVersion">1365357951589</long>
<long name="generation">198</long>
<arr name="filelist">
...
And with replication?command=details I also see the correct commit part as
above, BUT where the hell are the wrong info below the commit array are
coming from?
<str name="isMaster">true</str>
<str name="isSlave">false</str>
<long name="indexVersion">1365357774190</long>
<long name="generation">197</long>
The command replication?command=filelist&generation=197
replies with
<str name="status">invalid index generation</str>
Have a look into the sources:
Ahh, it is build in getReplicationDetails with:
details.add("isMaster", String.valueOf(isMaster));
details.add("isSlave", String.valueOf(isSlave));
long[] versionAndGeneration = getIndexVersion();
details.add("indexVersion", versionAndGeneration[0]);
details.add(GENERATION, versionAndGeneration[1]);
So getIndexVersion() gets a wrong version and generation, but why?
It first gets the searcher from the core and then tries to get
via the IndexReader the IndexCommit and then the commitData.
I think I should use remote debugging on master.
At least I now know that it is the master.
Regards
Bernd
Am 09.04.2013 08:35, schrieb Bernd Fehling:
> Hi Hoss,
>
> we don't use autoCommit and autoSoftCommit.
> We don't use openSearcher.
> We don't use transaction log.
>
> I can see it in the AdminGUI and with
> http://master_host:port/solr/replication?command=indexversion
>
> All files are replicated from master to slave, nothing lost.
> It is just that the gen/version differs and breaks our cronjobs which
> worked since solr 2.x.
> As you mentioned, it seams that the latest commit is fetched.
>
> Strange thing is,
> we start with a clean, empty index on master. With all commands we send
> a commit=true and, where applicable, an optimze=true.
> The master is always in state "optimzed and current" when replicating.
>
> How can it be that the searcher on master is referring to an older commit point
> if there is no such point. The logs show _AFTER_ the last optimize has finished
> a new searcher is started and the old one is closed.
>
> Also, we have replicateAfter startup, commit and optimize set but the AdminGUI
> and replication details only report replicateAfter commit and startup.
> Not really an error, but not that what is really set in the config!
>
> Very strange, I will try the patch.
>
> Regards
> Bernd
>
>
> Am 08.04.2013 20:12, schrieb Chris Hostetter:
>> : I know there was some effort to fix this but I must report
>> : that solr 4.2.1 has still problems with index version and
>> : index generation numbering in master/slave mode with replication.
>> ...
>> : RESULT: slave has different (higher) version number and is with generation 1 ahead :-(
>>
>> Can you please provide more details...
>>
>> * are you using autocommit? with what settings?
>> * are you using openSearcher=false in any of your commits?
>> * where exactly are you looking that you see the master/slave out of sync?
>> * are you observing any actual problems, or just seeing that the
>> gen/version are reported as different?
>>
>> As Joel mentioned, there is an open Jira related purely to the *display*
>> of information about gen/version between master & slave, because in many
>> cases the "searcher" in use on the master may refer to an older commit
>> point, but it doesn't mean there is any actual problem in replication --
>> the slave is still fetching & seraching the latest commit from the master
>> as intended....
>>
>> https://issues.apache.org/jira/browse/SOLR-4661
>>
>>
>> -Hoss
>>
--
*************************************************************
Bernd Fehling Bielefeld University Library
Dipl.-Inform. (FH) LibTec - Library Technology
Universitätsstr. 25 and Knowledge Management
33615 Bielefeld
Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************
Re: solr 4.2.1 still has problems with index version and index
generation
Posted by Bernd Fehling <be...@uni-bielefeld.de>.
Hi Hoss,
we don't use autoCommit and autoSoftCommit.
We don't use openSearcher.
We don't use transaction log.
I can see it in the AdminGUI and with
http://master_host:port/solr/replication?command=indexversion
All files are replicated from master to slave, nothing lost.
It is just that the gen/version differs and breaks our cronjobs which
worked since solr 2.x.
As you mentioned, it seams that the latest commit is fetched.
Strange thing is,
we start with a clean, empty index on master. With all commands we send
a commit=true and, where applicable, an optimze=true.
The master is always in state "optimzed and current" when replicating.
How can it be that the searcher on master is referring to an older commit point
if there is no such point. The logs show _AFTER_ the last optimize has finished
a new searcher is started and the old one is closed.
Also, we have replicateAfter startup, commit and optimize set but the AdminGUI
and replication details only report replicateAfter commit and startup.
Not really an error, but not that what is really set in the config!
Very strange, I will try the patch.
Regards
Bernd
Am 08.04.2013 20:12, schrieb Chris Hostetter:
> : I know there was some effort to fix this but I must report
> : that solr 4.2.1 has still problems with index version and
> : index generation numbering in master/slave mode with replication.
> ...
> : RESULT: slave has different (higher) version number and is with generation 1 ahead :-(
>
> Can you please provide more details...
>
> * are you using autocommit? with what settings?
> * are you using openSearcher=false in any of your commits?
> * where exactly are you looking that you see the master/slave out of sync?
> * are you observing any actual problems, or just seeing that the
> gen/version are reported as different?
>
> As Joel mentioned, there is an open Jira related purely to the *display*
> of information about gen/version between master & slave, because in many
> cases the "searcher" in use on the master may refer to an older commit
> point, but it doesn't mean there is any actual problem in replication --
> the slave is still fetching & seraching the latest commit from the master
> as intended....
>
> https://issues.apache.org/jira/browse/SOLR-4661
>
>
> -Hoss
>
Re: solr 4.2.1 still has problems with index version and index
generation
Posted by Chris Hostetter <ho...@fucit.org>.
: I know there was some effort to fix this but I must report
: that solr 4.2.1 has still problems with index version and
: index generation numbering in master/slave mode with replication.
...
: RESULT: slave has different (higher) version number and is with generation 1 ahead :-(
Can you please provide more details...
* are you using autocommit? with what settings?
* are you using openSearcher=false in any of your commits?
* where exactly are you looking that you see the master/slave out of sync?
* are you observing any actual problems, or just seeing that the
gen/version are reported as different?
As Joel mentioned, there is an open Jira related purely to the *display*
of information about gen/version between master & slave, because in many
cases the "searcher" in use on the master may refer to an older commit
point, but it doesn't mean there is any actual problem in replication --
the slave is still fetching & seraching the latest commit from the master
as intended....
https://issues.apache.org/jira/browse/SOLR-4661
-Hoss