You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Yatir Ben Shlomo <ya...@outbrain.com> on 2010/08/26 09:15:59 UTC

IllegalArgumentException excpetion : Path cannot be null 

I am running a zookeeper ensemble of 3 zookeeper instances
and established a solrCloud to work with it (2 masters , 2 slaves)
on one of the masters I keep noticing ZooKeeper related exceptions which I can't understand:
And the other is java.lang.IllegalArgumentException: Path cannot be null (PathUtils.java:45)

Here are my logs (I set the log level to FINE on zookeeper package)

 Anyone can identify the issue?
(I could not yet get any help from the solrCloud community)


FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null serverPath:null finished:false header:: -8,101  replyHeader:: -8,-1,0  request:: 30064776552,v{'/collections},v{},v{'/collections/ENPwl/shards/ENPWL1,'/collections/ENPwl/shards/ENPWL4,'/collections/ENPwl/shards/ENPWL2,'/collections,'/collections/ENPwl/shards/ENPWL3,'/collections/ENPwlMaster/shards/ENPWLMaster_3,'/collections/ENPwlMaster/shards/ENPWLMaster_4,'/live_nodes,'/collections/ENPwlMaster/shards/ENPWLMaster_1,'/collections/ENPwlMaster/shards/ENPWLMaster_2}  response:: null
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug
FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null serverPath:null finished:false header:: 540,8  replyHeader:: 540,-1,0  request:: '/collections,F  response:: v{'ENPwl,'ENPwlMaster}
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
SEVERE: Error while calling watcher
java.lang.IllegalArgumentException: Path cannot be null
        at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
        at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
        at org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL3 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process
INFO: Updating live nodes:org.apache.solr.common.cloud.SolrZkClient@55308275
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Updating live nodes from ZooKeeper...
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug
FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null serverPath:null finished:false header:: 541,8  replyHeader:: 541,-1,0  request:: '/live_nodes,F  response:: v{'ob1078.nydc1.outbrain.com:8983_solr2,'ob1078.nydc1.outbrain.com:8983_solr1,'ob1061.nydc1.outbrain.com:8983_solr2,'ob1062.nydc1.outbrain.com:8983_solr1,'ob1062.nydc1.outbrain.com:8983_solr2,'ob1061.nydc1.outbrain.com:8983_solr1,'ob1077.nydc1.outbrain.com:8983_solr2,'ob1077.nydc1.outbrain.com:8983_solr1}
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
SEVERE: Error while calling watcher
java.lang.IllegalArgumentException: Path cannot be null
        at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
        at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
        at org.apache.solr.cloud.ZkController$2.process(ZkController.java:321)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ConnectionManager process
INFO: Watcher org.apache.solr.common.cloud.ConnectionManager@339bb448 name:ZooKeeperConnection Watcher:zook1:2181,zook2:2181,zook3:2181 got event WatchedEvent: Server state change. New state: Disconnected path:null type:None
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_1 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_3 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL2 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_2 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_4 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$5 process
INFO: Detected a new or removed collection
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug
FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null serverPath:null finished:false header:: 542,8  replyHeader:: 542,-1,0  request:: '/collections,F  response:: v{'ENPwl,'ENPwlMaster}
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
SEVERE: Error while calling watcher
java.lang.IllegalArgumentException: Path cannot be null
        at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
        at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
        at org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL3 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process
INFO: Updating live nodes:org.apache.solr.common.cloud.SolrZkClient@55308275
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Updating live nodes from ZooKeeper...
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug
FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null serverPath:null finished:false header:: 543,8  replyHeader:: 543,-1,0  request:: '/live_nodes,F  response:: v{'ob1078.nydc1.outbrain.com:8983_solr2,'ob1078.nydc1.outbrain.com:8983_solr1,'ob1061.nydc1.outbrain.com:8983_solr2,'ob1062.nydc1.outbrain.com:8983_solr1,'ob1062.nydc1.outbrain.com:8983_solr2,'ob1061.nydc1.outbrain.com:8983_solr1,'ob1077.nydc1.outbrain.com:8983_solr2,'ob1077.nydc1.outbrain.com:8983_solr1}
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
SEVERE: Error while calling watcher
java.lang.IllegalArgumentException: Path cannot be null
        at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
        at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
        at org.apache.solr.cloud.ZkController$2.process(ZkController.java:321)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ConnectionManager process
INFO: Watcher org.apache.solr.common.cloud.ConnectionManager@339bb448 name:ZooKeeperConnection Watcher:zook1:2181,zook2:2181,zook3:2181 got event WatchedEvent: Server state change. New state: SyncConnected path:null type:None
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_1 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_3 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL2 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category info
INFO: Attempting connection to server zook2/192.168.252.62:2181
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category info
INFO: Priming connection to java.nio.channels.SocketChannel[connected local=/192.168.252.77:54892 remote=zook2/192.168.252.62:2181]
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category info
INFO: Server connection successful
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
SEVERE: Error while calling watcher
java.lang.IllegalArgumentException: Path cannot be null
        at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
        at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
        at org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_2 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process
INFO: Updating live nodes:org.apache.solr.common.cloud.SolrZkClient@29d03e78
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Updating live nodes from ZooKeeper...
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
SEVERE: Error while calling watcher
java.lang.IllegalArgumentException: Path cannot be null
        at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
        at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
        at org.apache.solr.cloud.ZkController$2.process(ZkController.java:321)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_4 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_1 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL3 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL2 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ConnectionManager process
INFO: Watcher org.apache.solr.common.cloud.ConnectionManager@40bb2bc3 name:ZooKeeperConnection Watcher:zook1:2181,zook2:2181,zook3:2181 got event WatchedEvent: Server state change. New state: SyncConnected path:null type:None
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_3 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$5 process
INFO: Detected a new or removed collection
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
SEVERE: Error while calling watcher
java.lang.IllegalArgumentException: Path cannot be null
        at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
        at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
        at org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_2 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process
INFO: Updating live nodes:org.apache.solr.common.cloud.SolrZkClient@29d03e78
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Updating live nodes from ZooKeeper...
Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
SEVERE: Error while calling watcher
java.lang.IllegalArgumentException: Path cannot be null
        at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
        at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
        at org.apache.solr.cloud.ZkController$2.process(ZkController.java:321)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_4 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process
INFO: Detected a shard change under ShardId:ENPWLMaster_1 in collection:ENPwlMaster
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$6 process
INFO: Notified of CloudState change
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Cloud state update for ZooKeeper already scheduled
Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process
INFO: Updating live nodes:org.apache.solr.common.cloud.SolrZkClient@29d03e78
Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState
INFO: Updating live nodes from ZooKeeper...
Aug 25, 2010 5:18:22 AM org.apache.log4j.Category debug
FINE: Got ping response for sessionid:0x12a97312613010b after 0ms

Re: IllegalArgumentException excpetion : Path cannot be null

Posted by Patrick Hunt <ph...@apache.org>.
The client (solr in this case) is passing a null path to the
ZooKeeper.getChildren(path, ... ) call.

java.lang.IllegalArgumentException: Path cannot be null
       at
org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
       at
org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
       at
org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)

I'm afraid you'll have to work with the solr team to determine the cause of
this.

Patrick

On Thu, Aug 26, 2010 at 12:15 AM, Yatir Ben Shlomo <ya...@outbrain.com>wrote:

> I am running a zookeeper ensemble of 3 zookeeper instances
> and established a solrCloud to work with it (2 masters , 2 slaves)
> on one of the masters I keep noticing ZooKeeper related exceptions which I
> can't understand:
> And the other is java.lang.IllegalArgumentException: Path cannot be null
> (PathUtils.java:45)
>
> Here are my logs (I set the log level to FINE on zookeeper package)
>
>  Anyone can identify the issue?
> (I could not yet get any help from the solrCloud community)
>
>
> FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null
> serverPath:null finished:false header:: -8,101  replyHeader:: -8,-1,0
>  request::
> 30064776552,v{'/collections},v{},v{'/collections/ENPwl/shards/ENPWL1,'/collections/ENPwl/shards/ENPWL4,'/collections/ENPwl/shards/ENPWL2,'/collections,'/collections/ENPwl/shards/ENPWL3,'/collections/ENPwlMaster/shards/ENPWLMaster_3,'/collections/ENPwlMaster/shards/ENPWLMaster_4,'/live_nodes,'/collections/ENPwlMaster/shards/ENPWLMaster_1,'/collections/ENPwlMaster/shards/ENPWLMaster_2}
>  response:: null
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug
> FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null
> serverPath:null finished:false header:: 540,8  replyHeader:: 540,-1,0
>  request:: '/collections,F  response:: v{'ENPwl,'ENPwlMaster}
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
> SEVERE: Error while calling watcher
> java.lang.IllegalArgumentException: Path cannot be null
>        at
> org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
>        at
> org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
>        at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
>        at
> org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315)
>        at
> org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL3 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process
> INFO: Updating live
> nodes:org.apache.solr.common.cloud.SolrZkClient@55308275
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Updating live nodes from ZooKeeper...
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug
> FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null
> serverPath:null finished:false header:: 541,8  replyHeader:: 541,-1,0
>  request:: '/live_nodes,F  response:: v{'ob1078.nydc1.outbrain.com:8983
> _solr2,'ob1078.nydc1.outbrain.com:8983
> _solr1,'ob1061.nydc1.outbrain.com:8983
> _solr2,'ob1062.nydc1.outbrain.com:8983
> _solr1,'ob1062.nydc1.outbrain.com:8983
> _solr2,'ob1061.nydc1.outbrain.com:8983
> _solr1,'ob1077.nydc1.outbrain.com:8983
> _solr2,'ob1077.nydc1.outbrain.com:8983_solr1}
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
> SEVERE: Error while calling watcher
> java.lang.IllegalArgumentException: Path cannot be null
>        at
> org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
>        at
> org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
>        at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
>        at
> org.apache.solr.cloud.ZkController$2.process(ZkController.java:321)
>        at
> org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ConnectionManager
> process
> INFO: Watcher org.apache.solr.common.cloud.ConnectionManager@339bb448name:ZooKeeperConnection Watcher:zook1:2181,zook2:2181,zook3:2181 got event
> WatchedEvent: Server state change. New state: Disconnected path:null
> type:None
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_1 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_3 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL2 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_2 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_4 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$5
> process
> INFO: Detected a new or removed collection
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug
> FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null
> serverPath:null finished:false header:: 542,8  replyHeader:: 542,-1,0
>  request:: '/collections,F  response:: v{'ENPwl,'ENPwlMaster}
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
> SEVERE: Error while calling watcher
> java.lang.IllegalArgumentException: Path cannot be null
>        at
> org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
>        at
> org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
>        at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
>        at
> org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315)
>        at
> org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL3 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process
> INFO: Updating live
> nodes:org.apache.solr.common.cloud.SolrZkClient@55308275
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Updating live nodes from ZooKeeper...
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug
> FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null
> serverPath:null finished:false header:: 543,8  replyHeader:: 543,-1,0
>  request:: '/live_nodes,F  response:: v{'ob1078.nydc1.outbrain.com:8983
> _solr2,'ob1078.nydc1.outbrain.com:8983
> _solr1,'ob1061.nydc1.outbrain.com:8983
> _solr2,'ob1062.nydc1.outbrain.com:8983
> _solr1,'ob1062.nydc1.outbrain.com:8983
> _solr2,'ob1061.nydc1.outbrain.com:8983
> _solr1,'ob1077.nydc1.outbrain.com:8983
> _solr2,'ob1077.nydc1.outbrain.com:8983_solr1}
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
> SEVERE: Error while calling watcher
> java.lang.IllegalArgumentException: Path cannot be null
>        at
> org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
>        at
> org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
>        at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
>        at
> org.apache.solr.cloud.ZkController$2.process(ZkController.java:321)
>        at
> org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ConnectionManager
> process
> INFO: Watcher org.apache.solr.common.cloud.ConnectionManager@339bb448name:ZooKeeperConnection Watcher:zook1:2181,zook2:2181,zook3:2181 got event
> WatchedEvent: Server state change. New state: SyncConnected path:null
> type:None
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_1 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_3 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL2 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category info
> INFO: Attempting connection to server zook2/192.168.252.62:2181
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category info
> INFO: Priming connection to java.nio.channels.SocketChannel[connected
> local=/192.168.252.77:54892 remote=zook2/192.168.252.62:2181]
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category info
> INFO: Server connection successful
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
> SEVERE: Error while calling watcher
> java.lang.IllegalArgumentException: Path cannot be null
>        at
> org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
>        at
> org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
>        at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
>        at
> org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315)
>        at
> org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_2 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process
> INFO: Updating live
> nodes:org.apache.solr.common.cloud.SolrZkClient@29d03e78
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Updating live nodes from ZooKeeper...
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
> SEVERE: Error while calling watcher
> java.lang.IllegalArgumentException: Path cannot be null
>        at
> org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
>        at
> org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
>        at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
>        at
> org.apache.solr.cloud.ZkController$2.process(ZkController.java:321)
>        at
> org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_4 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_1 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL3 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL2 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ConnectionManager
> process
> INFO: Watcher org.apache.solr.common.cloud.ConnectionManager@40bb2bc3name:ZooKeeperConnection Watcher:zook1:2181,zook2:2181,zook3:2181 got event
> WatchedEvent: Server state change. New state: SyncConnected path:null
> type:None
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_3 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$5
> process
> INFO: Detected a new or removed collection
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
> SEVERE: Error while calling watcher
> java.lang.IllegalArgumentException: Path cannot be null
>        at
> org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
>        at
> org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
>        at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
>        at
> org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315)
>        at
> org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_2 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process
> INFO: Updating live
> nodes:org.apache.solr.common.cloud.SolrZkClient@29d03e78
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Updating live nodes from ZooKeeper...
> Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error
> SEVERE: Error while calling watcher
> java.lang.IllegalArgumentException: Path cannot be null
>        at
> org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45)
>        at
> org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196)
>        at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200)
>        at
> org.apache.solr.cloud.ZkController$2.process(ZkController.java:321)
>        at
> org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425)
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_4 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4
> process
> INFO: Detected a shard change under ShardId:ENPWLMaster_1 in
> collection:ENPwlMaster
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$6
> process
> INFO: Notified of CloudState change
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Cloud state update for ZooKeeper already scheduled
> Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process
> INFO: Updating live
> nodes:org.apache.solr.common.cloud.SolrZkClient@29d03e78
> Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader
> updateCloudState
> INFO: Updating live nodes from ZooKeeper...
> Aug 25, 2010 5:18:22 AM org.apache.log4j.Category debug
> FINE: Got ping response for sessionid:0x12a97312613010b after 0ms
>