Indexing getting failed after some millions of documents

[Tim Burner <im...@gmail.com>]

Hi Everyone,

I have installed Solr-4.6 Cloud with external Zookeeper-3.4.5 and Tomcat-7,
the configuration is as mentioned below.

Single Machine Cluster Setup with 3 shards and 2 Replica deployed on 3
Tomcats with 3 Zookeeper.

Everything is installed good and fine, I start with the index and till I
reach some millions of documents(~1.6M) the indexing stops saying "*#503
Service Unavailable" *and the Cloud Dashboard log says

*"ERROR DistributedUpdateProcessor ClusterState says we are the leader,​
but locally we don't think so"*


*"ERROR SolrCore org.apache.solr.common.SolrException: ClusterState says we
are the leader (http://host:port1/solr/recollection_shard1_replica1),​ but
locally we don't think so. Request came from
http://host:port2/solr/recollection_shard2_replica1/"*


*"ERROR ZkController Error registering
SolrCore:org.apache.solr.common.SolrException: Error getting leader from zk
for shard shard2"*

Any suggestions/advice would be appreciated!

Thanks!
Tim

Re: Indexing getting failed after some millions of documents

[Erick Erickson <er...@gmail.com>]

How much memory have you allocated the JVMs? Also, what's does the
Solr log show on the machine that isn't coming up? Sounds like the
node went down and perhaps went into recovery....

And how are you indexing?

Best,
Erick

On Tue, May 20, 2014 at 11:54 PM, Tim Burner <im...@gmail.com> wrote:
> Hi Everyone,
>
> I have installed Solr-4.6 Cloud with external Zookeeper-3.4.5 and Tomcat-7,
> the configuration is as mentioned below.
>
> Single Machine Cluster Setup with 3 shards and 2 Replica deployed on 3
> Tomcats with 3 Zookeeper.
>
> Everything is installed good and fine, I start with the index and till I
> reach some millions of documents(~1.6M) the indexing stops saying "*#503
> Service Unavailable" *and the Cloud Dashboard log says
>
> *"ERROR DistributedUpdateProcessor ClusterState says we are the leader,
> but locally we don't think so"*
>
>
> *"ERROR SolrCore org.apache.solr.common.SolrException: ClusterState says we
> are the leader (http://host:port1/solr/recollection_shard1_replica1), but
> locally we don't think so. Request came from
> http://host:port2/solr/recollection_shard2_replica1/"*
>
>
> *"ERROR ZkController Error registering
> SolrCore:org.apache.solr.common.SolrException: Error getting leader from zk
> for shard shard2"*
>
> Any suggestions/advice would be appreciated!
>
> Thanks!
> Tim