You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by lancasp22 <pe...@findmypast.com> on 2017/08/08 16:49:21 UTC
SolrCloud - leader updates not updating followers
Hi,
I've recently created a solr cloud on solr 5.5.2 with a separate zookeeper
cluster. I write to the cloud by posting to update/json and the documents
appear fine in the leader.
The problem I have is that new documents added to the cloud aren't then
being automatically applied to the followers of each leader. If I query a
core then I can get different counts depending on which version of the core
the query ran over and the solr admin statistics page confirms that the
followers have fewer documents and are behind the leader.
If I restart the solr core for a follower, it does recover quickly and
brings itself up-to-date with the leader. Looking at the logs for the
follower you see that on re-start it identifies the leader and gets the
changes from that leader.
I think when documents are added to the leader these should be pushed to the
followers so maybe it's this push process that isn't being triggered.
It’s quite possible that I’ve made a simple error in the set-up but I’m
baffled as to what it is. Please can anyone advise on any configuration that
I need to check that might be causing these symptoms.
Thanks in anticipation,
Peter Lancaster.
--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-leader-updates-not-updating-followers-tp4349618.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud - leader updates not updating followers
Posted by Erick Erickson <er...@gmail.com>.
How did you set up your cluster? I hope you just used the collections
create API.
If you tried to do this by adding individual cores via the core admin
APIs, I'd advise
you to delete them all and use the collections API. In fact you should use the
collections API for everything; the "core admin API" is more for stand-alone and
internal use.
OK, a couple of other random thoughts:
1> in the admin/cloud view you should have a
tree>>collections>>your_collection>>state.json
node. What's in it? If that node isn't there it's in:
tree>>clusterstate.json
2> Are you totally sure you're referencing the external ZooKeeper ensemble
in all your Solr JVMs? This one's tricky. It's vaguely possible that
you're referencing
the embedded ZK in some of your Solr nodes. If you didn't provide the ensemble's
address to your Solr instances correctly it _might_ account for this.
Wild stab in the dark here.
Best,
Erick
On Tue, Aug 8, 2017 at 12:45 PM, Peter Lancaster
<pe...@findmypast.com> wrote:
> Hi Erik,
>
> Thanks for your quick reply. It's given me a few things to research and work on tomorrow.
>
> In the meantime, in case it triggers any other thoughts, just to say that our AutoCommit settings are
> <autoCommit>
> <maxDocs>1800000</maxDocs>
> <maxTime>300000</maxTime>
> </autoCommit>
> <autoSoftCommit>
> <maxTime>10000</maxTime>
> </autoSoftCommit>
>
> When I ingest data I don't see data appearing on the follower in the log.
>
> It really seems like the data isn't being sent from the leader. As I said it could easily be something stupid that I've done along the way but I can't see what it is.
>
> Thanks again,
> Peter.
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: 08 August 2017 18:23
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: SolrCloud - leader updates not updating followers
>
> This _better_ be a problem with your configuration or all my assumptions are false ;)
>
> What are you autocommit settings? The documents should be forwarded to each replica from the leader during ingestion. However, they are not visible on the follower until a hard commit(openSearcher=true) or soft commit is triggered. Long blog on all this here:
>
> https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> An easy way to check that docs are being sent to the follower is to tail solr.log and send a doc to the leader. You should see the doc arrive at the follower. If you do see that then it's your autocommit settings.
>
> NOTE: Solr promises eventual consistency. Due to the fact that autocommits will execute at different wall-clock times on the leaders and followers you can be out of sync by up to your autocommit interval.
>
> You can force a commit by a URL like
> "http://solr:port/solr/collection/update?commit=true" that will commit on all replicas in a collection that may also be useful for seeing if it's an autocommit issue.
>
> Best,
> Erick
>
> On Tue, Aug 8, 2017 at 9:49 AM, lancasp22 <pe...@findmypast.com> wrote:
>> Hi,
>>
>> I've recently created a solr cloud on solr 5.5.2 with a separate
>> zookeeper cluster. I write to the cloud by posting to update/json and
>> the documents appear fine in the leader.
>>
>> The problem I have is that new documents added to the cloud aren't
>> then being automatically applied to the followers of each leader. If I
>> query a core then I can get different counts depending on which
>> version of the core the query ran over and the solr admin statistics
>> page confirms that the followers have fewer documents and are behind the leader.
>>
>> If I restart the solr core for a follower, it does recover quickly and
>> brings itself up-to-date with the leader. Looking at the logs for the
>> follower you see that on re-start it identifies the leader and gets
>> the changes from that leader.
>>
>> I think when documents are added to the leader these should be pushed
>> to the followers so maybe it's this push process that isn't being triggered.
>>
>> It’s quite possible that I’ve made a simple error in the set-up but
>> I’m baffled as to what it is. Please can anyone advise on any
>> configuration that I need to check that might be causing these symptoms.
>>
>> Thanks in anticipation,
>> Peter Lancaster.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/SolrCloud-leader-updates-not-updati
>> ng-followers-tp4349618.html Sent from the Solr - User mailing list
>> archive at Nabble.com.
> ________________________________
>
> This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of findmypast shall be understood as neither given nor endorsed by it.
> ________________________________
>
> __________________________________________________________________________
>
> This email has been checked for virus and other malicious content prior to leaving our network.
> __________________________________________________________________________
RE: SolrCloud - leader updates not updating followers
Posted by Peter Lancaster <pe...@findmypast.com>.
Hi Erik,
Thanks for your quick reply. It's given me a few things to research and work on tomorrow.
In the meantime, in case it triggers any other thoughts, just to say that our AutoCommit settings are
<autoCommit>
<maxDocs>1800000</maxDocs>
<maxTime>300000</maxTime>
</autoCommit>
<autoSoftCommit>
<maxTime>10000</maxTime>
</autoSoftCommit>
When I ingest data I don't see data appearing on the follower in the log.
It really seems like the data isn't being sent from the leader. As I said it could easily be something stupid that I've done along the way but I can't see what it is.
Thanks again,
Peter.
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: 08 August 2017 18:23
To: solr-user <so...@lucene.apache.org>
Subject: Re: SolrCloud - leader updates not updating followers
This _better_ be a problem with your configuration or all my assumptions are false ;)
What are you autocommit settings? The documents should be forwarded to each replica from the leader during ingestion. However, they are not visible on the follower until a hard commit(openSearcher=true) or soft commit is triggered. Long blog on all this here:
https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
An easy way to check that docs are being sent to the follower is to tail solr.log and send a doc to the leader. You should see the doc arrive at the follower. If you do see that then it's your autocommit settings.
NOTE: Solr promises eventual consistency. Due to the fact that autocommits will execute at different wall-clock times on the leaders and followers you can be out of sync by up to your autocommit interval.
You can force a commit by a URL like
"http://solr:port/solr/collection/update?commit=true" that will commit on all replicas in a collection that may also be useful for seeing if it's an autocommit issue.
Best,
Erick
On Tue, Aug 8, 2017 at 9:49 AM, lancasp22 <pe...@findmypast.com> wrote:
> Hi,
>
> I've recently created a solr cloud on solr 5.5.2 with a separate
> zookeeper cluster. I write to the cloud by posting to update/json and
> the documents appear fine in the leader.
>
> The problem I have is that new documents added to the cloud aren't
> then being automatically applied to the followers of each leader. If I
> query a core then I can get different counts depending on which
> version of the core the query ran over and the solr admin statistics
> page confirms that the followers have fewer documents and are behind the leader.
>
> If I restart the solr core for a follower, it does recover quickly and
> brings itself up-to-date with the leader. Looking at the logs for the
> follower you see that on re-start it identifies the leader and gets
> the changes from that leader.
>
> I think when documents are added to the leader these should be pushed
> to the followers so maybe it's this push process that isn't being triggered.
>
> It’s quite possible that I’ve made a simple error in the set-up but
> I’m baffled as to what it is. Please can anyone advise on any
> configuration that I need to check that might be causing these symptoms.
>
> Thanks in anticipation,
> Peter Lancaster.
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/SolrCloud-leader-updates-not-updati
> ng-followers-tp4349618.html Sent from the Solr - User mailing list
> archive at Nabble.com.
________________________________
This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of findmypast shall be understood as neither given nor endorsed by it.
________________________________
__________________________________________________________________________
This email has been checked for virus and other malicious content prior to leaving our network.
__________________________________________________________________________
Re: SolrCloud - leader updates not updating followers
Posted by Erick Erickson <er...@gmail.com>.
This _better_ be a problem with your configuration or all my
assumptions are false ;)
What are you autocommit settings? The documents should be forwarded to
each replica from the leader during ingestion. However, they are not
visible on the follower until a hard commit(openSearcher=true) or soft
commit is triggered. Long blog on all this here:
https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
An easy way to check that docs are being sent to the follower is to
tail solr.log and send a doc to the leader. You should see the doc
arrive at the follower. If you do see that then it's your autocommit
settings.
NOTE: Solr promises eventual consistency. Due to the fact that
autocommits will execute at different wall-clock times on the leaders
and followers you can be out of sync by up to your autocommit
interval.
You can force a commit by a URL like
"http://solr:port/solr/collection/update?commit=true" that will commit
on all replicas in a collection that may also be useful for seeing if
it's an autocommit issue.
Best,
Erick
On Tue, Aug 8, 2017 at 9:49 AM, lancasp22
<pe...@findmypast.com> wrote:
> Hi,
>
> I've recently created a solr cloud on solr 5.5.2 with a separate zookeeper
> cluster. I write to the cloud by posting to update/json and the documents
> appear fine in the leader.
>
> The problem I have is that new documents added to the cloud aren't then
> being automatically applied to the followers of each leader. If I query a
> core then I can get different counts depending on which version of the core
> the query ran over and the solr admin statistics page confirms that the
> followers have fewer documents and are behind the leader.
>
> If I restart the solr core for a follower, it does recover quickly and
> brings itself up-to-date with the leader. Looking at the logs for the
> follower you see that on re-start it identifies the leader and gets the
> changes from that leader.
>
> I think when documents are added to the leader these should be pushed to the
> followers so maybe it's this push process that isn't being triggered.
>
> It’s quite possible that I’ve made a simple error in the set-up but I’m
> baffled as to what it is. Please can anyone advise on any configuration that
> I need to check that might be causing these symptoms.
>
> Thanks in anticipation,
> Peter Lancaster.
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-leader-updates-not-updating-followers-tp4349618.html
> Sent from the Solr - User mailing list archive at Nabble.com.