You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by J Mohamed Zahoor <za...@indix.com> on 2013/04/16 18:49:39 UTC
zkState changes too often
Hi
I am using SolrCloud (4.1) with 6 nodes.
When i index the documents from the mapper and as the load increases.. i see these messages in my mapper logs…
WHich looks like it is slowing down my indexing speed.
2013-04-16 06:04:18,013 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (5)
2013-04-16 06:04:18,186 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (6)
2013-04-16 06:04:18,186 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (6)
2013-04-16 06:04:19,485 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
2013-04-16 06:04:19,487 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
2013-04-16 06:08:30,006 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
2013-04-16 06:08:30,010 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (5)
2013-04-16 06:08:30,010 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
2013-04-16 06:08:30,019 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (5)
2013-04-16 06:08:35,443 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
2013-04-16 06:08:35,446 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (6)
2013-04-16 06:08:35,446 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
2013-04-16 06:08:35,459 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (6)
2013-04-16 06:08:48,929 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
2013-04-16 06:08:48,931 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
2013-04-16 06:09:12,005 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
2013-04-16 06:09:12,010 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (5)
2013-04-16 06:09:12,011 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
2013-04-16 06:09:12,014 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (5)
2013-04-16 06:09:15,438 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
2013-04-16 06:09:15,441 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (6)
2013-04-16 06:09:15,441 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent stat
I tried increasing the Zk timeout from 15 to 20 sec… but i still see this message…
anything i might try to avoid this?
./Zahoor
Re: zkState changes too often
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi Mark;
I have same situation with my Solr 4.2.1 nodes. These are from my startup
parameters:
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
-XX:NewRatio=3 -XX:MaxTenuringThreshold=8 -XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled -XX:+UseLargePages -XX:+AggressiveOpts
My nodes are at Amazon here is a part from log messages:
INFO: A cluster state change: WatchedEvent state:SyncConnected
type:NodeDataChanged path:/clusterstate.json, has occurred - updating...
(live nodes size: 10)
Do you have an idea about it?
2013/4/18 Mark Miller <ma...@gmail.com>
>
> On Apr 18, 2013, at 8:40 AM, jmozah <jm...@gmail.com> wrote:
>
> >
> >
> > On 16-Apr-2013, at 11:16 PM, Mark Miller <ma...@gmail.com> wrote:
> >
> >> Are you using a the concurrent low pause garbage collector or perhaps
> G1?
> >
> >
> > I use the default one which comes in jdk 1.7.
>
> It varies by platform, but 99% that means you are using the throughput
> collector and you should try the CMS collector instead.
>
> - Mark
>
> >
> >>
> >> Are you able to use something like visualvm to pinpoint what the
> bottleneck might be?
> >
> > Unfortunately.. it is prod machine and i could not replicate it locally.
> >
> >>
> >> Otherwise, keep raising the timeout.
> >
> >
> > Thats what i did now.. will see if it comes in the next run..
> >
> > ./zahoor
> >
>
>
Re: zkState changes too often
Posted by Mark Miller <ma...@gmail.com>.
On Apr 18, 2013, at 8:40 AM, jmozah <jm...@gmail.com> wrote:
>
>
> On 16-Apr-2013, at 11:16 PM, Mark Miller <ma...@gmail.com> wrote:
>
>> Are you using a the concurrent low pause garbage collector or perhaps G1?
>
>
> I use the default one which comes in jdk 1.7.
It varies by platform, but 99% that means you are using the throughput collector and you should try the CMS collector instead.
- Mark
>
>>
>> Are you able to use something like visualvm to pinpoint what the bottleneck might be?
>
> Unfortunately.. it is prod machine and i could not replicate it locally.
>
>>
>> Otherwise, keep raising the timeout.
>
>
> Thats what i did now.. will see if it comes in the next run..
>
> ./zahoor
>
Re: zkState changes too often
Posted by jmozah <jm...@gmail.com>.
On 16-Apr-2013, at 11:16 PM, Mark Miller <ma...@gmail.com> wrote:
> Are you using a the concurrent low pause garbage collector or perhaps G1?
I use the default one which comes in jdk 1.7.
>
> Are you able to use something like visualvm to pinpoint what the bottleneck might be?
Unfortunately.. it is prod machine and i could not replicate it locally.
>
> Otherwise, keep raising the timeout.
Thats what i did now.. will see if it comes in the next run..
./zahoor
Re: zkState changes too often
Posted by Mark Miller <ma...@gmail.com>.
Are you using a the concurrent low pause garbage collector or perhaps G1?
Are you able to use something like visualvm to pinpoint what the bottleneck might be?
Otherwise, keep raising the timeout. This means Solr and Zk are not able to talk for that much time - either something needs to be tuned or the time allowed raised.
- Mark
On Apr 16, 2013, at 12:49 PM, J Mohamed Zahoor <za...@indix.com> wrote:
> Hi
>
> I am using SolrCloud (4.1) with 6 nodes.
> When i index the documents from the mapper and as the load increases.. i see these messages in my mapper logs…
> WHich looks like it is slowing down my indexing speed.
>
>
> 2013-04-16 06:04:18,013 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (5)
> 2013-04-16 06:04:18,186 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (6)
> 2013-04-16 06:04:18,186 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (6)
> 2013-04-16 06:04:19,485 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> 2013-04-16 06:04:19,487 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> 2013-04-16 06:08:30,006 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> 2013-04-16 06:08:30,010 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (5)
> 2013-04-16 06:08:30,010 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> 2013-04-16 06:08:30,019 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (5)
> 2013-04-16 06:08:35,443 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> 2013-04-16 06:08:35,446 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (6)
> 2013-04-16 06:08:35,446 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> 2013-04-16 06:08:35,459 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (6)
> 2013-04-16 06:08:48,929 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> 2013-04-16 06:08:48,931 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> 2013-04-16 06:09:12,005 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> 2013-04-16 06:09:12,010 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (5)
> 2013-04-16 06:09:12,011 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> 2013-04-16 06:09:12,014 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (5)
> 2013-04-16 06:09:15,438 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> 2013-04-16 06:09:15,441 INFO org.apache.solr.common.cloud.ZkStateReader: Updating live nodes... (6)
> 2013-04-16 06:09:15,441 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent stat
>
> I tried increasing the Zk timeout from 15 to 20 sec… but i still see this message…
> anything i might try to avoid this?
>
> ./Zahoor
>
>