You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Piyush Kunal <pi...@myntra.com> on 2016/12/15 11:03:54 UTC
Getting Error - Session expired for /collections/sprod/state.json
- We have solr6.1.0 cluster running on production with 1 shard and 5
replicas.
- Zookeeper quorum on 3 nodes.
- Using a chroot in zookeeper to segregate the configs from other
collections.
- Using solrj5.1.0 as our client to query solr.
Usually things work fine but on and off we witness this exception coming up:
=============================================================
org.apache.solr.common.SolrException: Could not load collection from
ZK:sprod
at
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:815)
at
org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateReader.java:477)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1174)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:807)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:782)
--
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /collections/sprod/state.json
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
at
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308)
at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
at
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
--
org.apache.solr.common.SolrException: Could not load collection from
ZK:sprod
at
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:815)
at
org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateReader.java:477)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1174)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:807)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:782)
--
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /collections/sprod/state.json
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
at
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308)
at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
at
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
=============================================================
This is our zoo.cfg:
======================================
tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=192.168.70.27:2888:3888
server.2=192.168.70.64:2889:3889
server.3=192.168.70.26:2889:3889
maxClientCnxns=300
maxSessionTimeout=90000
=======================================
This is our solr.xml on server side
=======================================
<solr>
<solrcloud>
<str name="host">${host:}</str>
<int name="hostPort">${jetty.port:8983}</int>
<str name="hostContext">${hostContext:solr}</str>
<bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
<int name="zkClientTimeout">${zkClientTimeout:30000}</int>
<int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:600000}</int>
<int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:60000}</int>
<str name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}</str>
<str name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}</str>
</solrcloud>
<shardHandlerFactory name="shardHandlerFactory"
class="HttpShardHandlerFactory">
<int name="socketTimeout">${socketTimeout:600000}</int>
<int name="connTimeout">${connTimeout:60000}</int>
</shardHandlerFactory>
</solr>
=======================================
Any help appreciated.
Regards,
Piyush
Re: Getting Error - Session expired for /collections/sprod/state.json
Posted by Erick Erickson <er...@gmail.com>.
Look at your connection timeouts and your ZK timeouts.
This usually means your Solr instances are going
into heavy GC as Yago mentions. You can turn on
GC logging if it's not already then use something like
GCViewer to get a handle on the GC.
You really have two options:
1> if it is GC, tune your instances to avoid that if posisble.
This is "more art than science".
2> lengthen timeouts, there are a series of them for client
connections, Solr<->Solr connections and ZK<->Solr
connections....
Best,
Erick
On Fri, Dec 16, 2016 at 2:07 AM, Yago Riveiro <ya...@gmail.com> wrote:
> Do some gc profiling to get some information about. It's possible you have configure a small heap and you are running in gc stop the world issues.
>
> Normally zookeeper erros are bounded to gc and network latency issues
>
> --
>
> /Yago Riveiro
>
> On 16 Dec 2016, 09:49 +0000, Piyush Kunal <pi...@myntra.com>, wrote:
>> Looks like an issue with 6.x version then.
>> But this seems too basic. Not sure if community would not have caught this
>> till now.
>>
>> On Fri, Dec 16, 2016 at 2:55 PM, Yago Riveiro <yago.riveiro@gmail.com
>> wrote:
>>
>> > I had some of this error in my logs too on 6.3.0
>> >
>> > My cluster also index like 20K docs/sec I don't know why.
>> >
>> > --
>> >
>> > /Yago Riveiro
>> >
>> > On 16 Dec 2016, 08:39 +0000, Piyush Kunal <pi...@myntra.com>,
>> > wrote:
>> > > Anyone has noticed such issue before?
>> > >
>> > > On Thu, Dec 15, 2016 at 4:36 PM, Piyush Kunal <piyush.kunal@myntra.com
>> > > wrote:
>> > >
>> > > > This is happening when heavy indexing like 100/second is going on.
>> > > >
>> > > > On Thu, Dec 15, 2016 at 4:33 PM, Piyush Kunal <piyush.kunal@myntra.com
>> > > > wrote:
>> > > >
>> > > > > - We have solr6.1.0 cluster running on production with 1 shard and 5
>> > > > > replicas.
>> > > > > - Zookeeper quorum on 3 nodes.
>> > > > > - Using a chroot in zookeeper to segregate the configs from other
>> > > > > collections.
>> > > > > - Using solrj5.1.0 as our client to query solr.
>> > > > >
>> > > > >
>> > > > >
>> > > > > Usually things work fine but on and off we witness this exception
>> > coming
>> > > > > up:
>> > > > > =============================================================
>> > > > > org.apache.solr.common.SolrException: Could not load collection from
>> > > > > ZK:sprod
>> > > > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
>> > > > > (ZkStateReader.java:815)
>> > > > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
>> > > > > er.java:477)
>> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
>> > > > > ection(CloudSolrClient.java:1174)
>> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
>> > > > > hRetryOnStaleState(CloudSolrClient.java:807)
>> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
>> > > > > oudSolrClient.java:782)
>> > > > > --
>> > > > > Caused by: org.apache.zookeeper.KeeperException$
>> > SessionExpiredException:
>> > > > > KeeperErrorCode = Session expired for /collections/sprod/state.json
>> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
>> > > > > java:127)
>> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
>> > > > > java:51)
>> > > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> > > > > ient.java:311)
>> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> > > > > ient.java:308)
>> > > > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
>> > > > > CmdExecutor.java:61)
>> > > > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
>> > > > > t.java:308)
>> > > > > --
>> > > > > org.apache.solr.common.SolrException: Could not load collection from
>> > > > > ZK:sprod
>> > > > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
>> > > > > (ZkStateReader.java:815)
>> > > > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
>> > > > > er.java:477)
>> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
>> > > > > ection(CloudSolrClient.java:1174)
>> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
>> > > > > hRetryOnStaleState(CloudSolrClient.java:807)
>> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
>> > > > > oudSolrClient.java:782)
>> > > > > --
>> > > > > Caused by: org.apache.zookeeper.KeeperException$
>> > SessionExpiredException:
>> > > > > KeeperErrorCode = Session expired for /collections/sprod/state.json
>> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
>> > > > > java:127)
>> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
>> > > > > java:51)
>> > > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> > > > > ient.java:311)
>> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> > > > > ient.java:308)
>> > > > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
>> > > > > CmdExecutor.java:61)
>> > > > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
>> > > > > t.java:308)
>> > > > > =============================================================
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > This is our zoo.cfg:
>> > > > > ======================================
>> > > > > tickTime=2000
>> > > > > dataDir=/var/lib/zookeeper
>> > > > > clientPort=2181
>> > > > > initLimit=5
>> > > > > syncLimit=2
>> > > > > server.1=192.168.70.27:2888:3888
>> > > > > server.2=192.168.70.64:2889:3889
>> > > > > server.3=192.168.70.26:2889:3889
>> > > > > maxClientCnxns=300
>> > > > > maxSessionTimeout=90000
>> > > > > =======================================
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > This is our solr.xml on server side
>> > > > > =======================================
>> > > > >
>> > > > > <solr
>> > > > >
>> > > > > <solrcloud
>> > > > >
>> > > > > <str name="host">${host:}</str
>> > > > > <int name="hostPort">${jetty.port:8983}</int
>> > > > > <str name="hostContext">${hostContext:solr}</str
>> > > > >
>> > > > > <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool
>> > > > >
>> > > > > <int name="zkClientTimeout">${zkClientTimeout:30000}</int
>> > > > > <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:
>> > 600000}</int
>> > > > > <int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:
>> > 60000}</int
>> > > > > <str name="zkCredentialsProvider">${zkCredentialsProvider:org.
>> > apache.solr.common.cloud.DefaultZkCredentialsProvider}</str
>> > > > > <str name="zkACLProvider">${zkACLProvider:org.apache.solr.
>> > common.cloud.DefaultZkACLProvider}</str
>> > > > >
>> > > > > </solrcloud
>> > > > >
>> > > > > <shardHandlerFactory name="shardHandlerFactory"
>> > > > > class="HttpShardHandlerFactory"
>> > > > > <int name="socketTimeout">${socketTimeout:600000}</int
>> > > > > <int name="connTimeout">${connTimeout:60000}</int
>> > > > > </shardHandlerFactory
>> > > > > </solr
>> > > > >
>> > > > > =======================================
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > Any help appreciated.
>> > > > >
>> > > > > Regards,
>> > > > > Piyush
>> > > > >
>> > > >
>> > > >
>> >
Re: Getting Error - Session expired for
/collections/sprod/state.json
Posted by Yago Riveiro <ya...@gmail.com>.
Do some gc profiling to get some information about. It's possible you have configure a small heap and you are running in gc stop the world issues.
Normally zookeeper erros are bounded to gc and network latency issues
--
/Yago Riveiro
On 16 Dec 2016, 09:49 +0000, Piyush Kunal <pi...@myntra.com>, wrote:
> Looks like an issue with 6.x version then.
> But this seems too basic. Not sure if community would not have caught this
> till now.
>
> On Fri, Dec 16, 2016 at 2:55 PM, Yago Riveiro <yago.riveiro@gmail.com
> wrote:
>
> > I had some of this error in my logs too on 6.3.0
> >
> > My cluster also index like 20K docs/sec I don't know why.
> >
> > --
> >
> > /Yago Riveiro
> >
> > On 16 Dec 2016, 08:39 +0000, Piyush Kunal <pi...@myntra.com>,
> > wrote:
> > > Anyone has noticed such issue before?
> > >
> > > On Thu, Dec 15, 2016 at 4:36 PM, Piyush Kunal <piyush.kunal@myntra.com
> > > wrote:
> > >
> > > > This is happening when heavy indexing like 100/second is going on.
> > > >
> > > > On Thu, Dec 15, 2016 at 4:33 PM, Piyush Kunal <piyush.kunal@myntra.com
> > > > wrote:
> > > >
> > > > > - We have solr6.1.0 cluster running on production with 1 shard and 5
> > > > > replicas.
> > > > > - Zookeeper quorum on 3 nodes.
> > > > > - Using a chroot in zookeeper to segregate the configs from other
> > > > > collections.
> > > > > - Using solrj5.1.0 as our client to query solr.
> > > > >
> > > > >
> > > > >
> > > > > Usually things work fine but on and off we witness this exception
> > coming
> > > > > up:
> > > > > =============================================================
> > > > > org.apache.solr.common.SolrException: Could not load collection from
> > > > > ZK:sprod
> > > > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> > > > > (ZkStateReader.java:815)
> > > > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
> > > > > er.java:477)
> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
> > > > > ection(CloudSolrClient.java:1174)
> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
> > > > > hRetryOnStaleState(CloudSolrClient.java:807)
> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
> > > > > oudSolrClient.java:782)
> > > > > --
> > > > > Caused by: org.apache.zookeeper.KeeperException$
> > SessionExpiredException:
> > > > > KeeperErrorCode = Session expired for /collections/sprod/state.json
> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > > java:127)
> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > > java:51)
> > > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > > ient.java:311)
> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > > ient.java:308)
> > > > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
> > > > > CmdExecutor.java:61)
> > > > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
> > > > > t.java:308)
> > > > > --
> > > > > org.apache.solr.common.SolrException: Could not load collection from
> > > > > ZK:sprod
> > > > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> > > > > (ZkStateReader.java:815)
> > > > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
> > > > > er.java:477)
> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
> > > > > ection(CloudSolrClient.java:1174)
> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
> > > > > hRetryOnStaleState(CloudSolrClient.java:807)
> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
> > > > > oudSolrClient.java:782)
> > > > > --
> > > > > Caused by: org.apache.zookeeper.KeeperException$
> > SessionExpiredException:
> > > > > KeeperErrorCode = Session expired for /collections/sprod/state.json
> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > > java:127)
> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > > java:51)
> > > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > > ient.java:311)
> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > > ient.java:308)
> > > > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
> > > > > CmdExecutor.java:61)
> > > > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
> > > > > t.java:308)
> > > > > =============================================================
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > This is our zoo.cfg:
> > > > > ======================================
> > > > > tickTime=2000
> > > > > dataDir=/var/lib/zookeeper
> > > > > clientPort=2181
> > > > > initLimit=5
> > > > > syncLimit=2
> > > > > server.1=192.168.70.27:2888:3888
> > > > > server.2=192.168.70.64:2889:3889
> > > > > server.3=192.168.70.26:2889:3889
> > > > > maxClientCnxns=300
> > > > > maxSessionTimeout=90000
> > > > > =======================================
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > This is our solr.xml on server side
> > > > > =======================================
> > > > >
> > > > > <solr
> > > > >
> > > > > <solrcloud
> > > > >
> > > > > <str name="host">${host:}</str
> > > > > <int name="hostPort">${jetty.port:8983}</int
> > > > > <str name="hostContext">${hostContext:solr}</str
> > > > >
> > > > > <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool
> > > > >
> > > > > <int name="zkClientTimeout">${zkClientTimeout:30000}</int
> > > > > <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:
> > 600000}</int
> > > > > <int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:
> > 60000}</int
> > > > > <str name="zkCredentialsProvider">${zkCredentialsProvider:org.
> > apache.solr.common.cloud.DefaultZkCredentialsProvider}</str
> > > > > <str name="zkACLProvider">${zkACLProvider:org.apache.solr.
> > common.cloud.DefaultZkACLProvider}</str
> > > > >
> > > > > </solrcloud
> > > > >
> > > > > <shardHandlerFactory name="shardHandlerFactory"
> > > > > class="HttpShardHandlerFactory"
> > > > > <int name="socketTimeout">${socketTimeout:600000}</int
> > > > > <int name="connTimeout">${connTimeout:60000}</int
> > > > > </shardHandlerFactory
> > > > > </solr
> > > > >
> > > > > =======================================
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Any help appreciated.
> > > > >
> > > > > Regards,
> > > > > Piyush
> > > > >
> > > >
> > > >
> >
Re: Getting Error - Session expired for /collections/sprod/state.json
Posted by Piyush Kunal <pi...@myntra.com>.
Looks like an issue with 6.x version then.
But this seems too basic. Not sure if community would not have caught this
till now.
On Fri, Dec 16, 2016 at 2:55 PM, Yago Riveiro <ya...@gmail.com>
wrote:
> I had some of this error in my logs too on 6.3.0
>
> My cluster also index like 20K docs/sec I don't know why.
>
> --
>
> /Yago Riveiro
>
> On 16 Dec 2016, 08:39 +0000, Piyush Kunal <pi...@myntra.com>,
> wrote:
> > Anyone has noticed such issue before?
> >
> > On Thu, Dec 15, 2016 at 4:36 PM, Piyush Kunal <piyush.kunal@myntra.com
> > wrote:
> >
> > > This is happening when heavy indexing like 100/second is going on.
> > >
> > > On Thu, Dec 15, 2016 at 4:33 PM, Piyush Kunal <piyush.kunal@myntra.com
> > > wrote:
> > >
> > > > - We have solr6.1.0 cluster running on production with 1 shard and 5
> > > > replicas.
> > > > - Zookeeper quorum on 3 nodes.
> > > > - Using a chroot in zookeeper to segregate the configs from other
> > > > collections.
> > > > - Using solrj5.1.0 as our client to query solr.
> > > >
> > > >
> > > >
> > > > Usually things work fine but on and off we witness this exception
> coming
> > > > up:
> > > > =============================================================
> > > > org.apache.solr.common.SolrException: Could not load collection from
> > > > ZK:sprod
> > > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> > > > (ZkStateReader.java:815)
> > > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
> > > > er.java:477)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
> > > > ection(CloudSolrClient.java:1174)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
> > > > hRetryOnStaleState(CloudSolrClient.java:807)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
> > > > oudSolrClient.java:782)
> > > > --
> > > > Caused by: org.apache.zookeeper.KeeperException$
> SessionExpiredException:
> > > > KeeperErrorCode = Session expired for /collections/sprod/state.json
> > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > java:127)
> > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > java:51)
> > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > ient.java:311)
> > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > ient.java:308)
> > > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
> > > > CmdExecutor.java:61)
> > > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
> > > > t.java:308)
> > > > --
> > > > org.apache.solr.common.SolrException: Could not load collection from
> > > > ZK:sprod
> > > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> > > > (ZkStateReader.java:815)
> > > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
> > > > er.java:477)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
> > > > ection(CloudSolrClient.java:1174)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
> > > > hRetryOnStaleState(CloudSolrClient.java:807)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
> > > > oudSolrClient.java:782)
> > > > --
> > > > Caused by: org.apache.zookeeper.KeeperException$
> SessionExpiredException:
> > > > KeeperErrorCode = Session expired for /collections/sprod/state.json
> > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > java:127)
> > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > java:51)
> > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > ient.java:311)
> > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > ient.java:308)
> > > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
> > > > CmdExecutor.java:61)
> > > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
> > > > t.java:308)
> > > > =============================================================
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > This is our zoo.cfg:
> > > > ======================================
> > > > tickTime=2000
> > > > dataDir=/var/lib/zookeeper
> > > > clientPort=2181
> > > > initLimit=5
> > > > syncLimit=2
> > > > server.1=192.168.70.27:2888:3888
> > > > server.2=192.168.70.64:2889:3889
> > > > server.3=192.168.70.26:2889:3889
> > > > maxClientCnxns=300
> > > > maxSessionTimeout=90000
> > > > =======================================
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > This is our solr.xml on server side
> > > > =======================================
> > > >
> > > > <solr
> > > >
> > > > <solrcloud
> > > >
> > > > <str name="host">${host:}</str
> > > > <int name="hostPort">${jetty.port:8983}</int
> > > > <str name="hostContext">${hostContext:solr}</str
> > > >
> > > > <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool
> > > >
> > > > <int name="zkClientTimeout">${zkClientTimeout:30000}</int
> > > > <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:
> 600000}</int
> > > > <int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:
> 60000}</int
> > > > <str name="zkCredentialsProvider">${zkCredentialsProvider:org.
> apache.solr.common.cloud.DefaultZkCredentialsProvider}</str
> > > > <str name="zkACLProvider">${zkACLProvider:org.apache.solr.
> common.cloud.DefaultZkACLProvider}</str
> > > >
> > > > </solrcloud
> > > >
> > > > <shardHandlerFactory name="shardHandlerFactory"
> > > > class="HttpShardHandlerFactory"
> > > > <int name="socketTimeout">${socketTimeout:600000}</int
> > > > <int name="connTimeout">${connTimeout:60000}</int
> > > > </shardHandlerFactory
> > > > </solr
> > > >
> > > > =======================================
> > > >
> > > >
> > > >
> > > >
> > > > Any help appreciated.
> > > >
> > > > Regards,
> > > > Piyush
> > > >
> > >
> > >
>
Re: Getting Error - Session expired for
/collections/sprod/state.json
Posted by Yago Riveiro <ya...@gmail.com>.
I had some of this error in my logs too on 6.3.0
My cluster also index like 20K docs/sec I don't know why.
--
/Yago Riveiro
On 16 Dec 2016, 08:39 +0000, Piyush Kunal <pi...@myntra.com>, wrote:
> Anyone has noticed such issue before?
>
> On Thu, Dec 15, 2016 at 4:36 PM, Piyush Kunal <piyush.kunal@myntra.com
> wrote:
>
> > This is happening when heavy indexing like 100/second is going on.
> >
> > On Thu, Dec 15, 2016 at 4:33 PM, Piyush Kunal <piyush.kunal@myntra.com
> > wrote:
> >
> > > - We have solr6.1.0 cluster running on production with 1 shard and 5
> > > replicas.
> > > - Zookeeper quorum on 3 nodes.
> > > - Using a chroot in zookeeper to segregate the configs from other
> > > collections.
> > > - Using solrj5.1.0 as our client to query solr.
> > >
> > >
> > >
> > > Usually things work fine but on and off we witness this exception coming
> > > up:
> > > =============================================================
> > > org.apache.solr.common.SolrException: Could not load collection from
> > > ZK:sprod
> > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> > > (ZkStateReader.java:815)
> > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
> > > er.java:477)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
> > > ection(CloudSolrClient.java:1174)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
> > > hRetryOnStaleState(CloudSolrClient.java:807)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
> > > oudSolrClient.java:782)
> > > --
> > > Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /collections/sprod/state.json
> > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > java:127)
> > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > java:51)
> > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > ient.java:311)
> > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > ient.java:308)
> > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
> > > CmdExecutor.java:61)
> > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
> > > t.java:308)
> > > --
> > > org.apache.solr.common.SolrException: Could not load collection from
> > > ZK:sprod
> > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> > > (ZkStateReader.java:815)
> > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
> > > er.java:477)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
> > > ection(CloudSolrClient.java:1174)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
> > > hRetryOnStaleState(CloudSolrClient.java:807)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
> > > oudSolrClient.java:782)
> > > --
> > > Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /collections/sprod/state.json
> > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > java:127)
> > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > java:51)
> > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > ient.java:311)
> > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > ient.java:308)
> > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
> > > CmdExecutor.java:61)
> > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
> > > t.java:308)
> > > =============================================================
> > >
> > >
> > >
> > >
> > >
> > > This is our zoo.cfg:
> > > ======================================
> > > tickTime=2000
> > > dataDir=/var/lib/zookeeper
> > > clientPort=2181
> > > initLimit=5
> > > syncLimit=2
> > > server.1=192.168.70.27:2888:3888
> > > server.2=192.168.70.64:2889:3889
> > > server.3=192.168.70.26:2889:3889
> > > maxClientCnxns=300
> > > maxSessionTimeout=90000
> > > =======================================
> > >
> > >
> > >
> > >
> > >
> > > This is our solr.xml on server side
> > > =======================================
> > >
> > > <solr
> > >
> > > <solrcloud
> > >
> > > <str name="host">${host:}</str
> > > <int name="hostPort">${jetty.port:8983}</int
> > > <str name="hostContext">${hostContext:solr}</str
> > >
> > > <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool
> > >
> > > <int name="zkClientTimeout">${zkClientTimeout:30000}</int
> > > <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:600000}</int
> > > <int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:60000}</int
> > > <str name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}</str
> > > <str name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}</str
> > >
> > > </solrcloud
> > >
> > > <shardHandlerFactory name="shardHandlerFactory"
> > > class="HttpShardHandlerFactory"
> > > <int name="socketTimeout">${socketTimeout:600000}</int
> > > <int name="connTimeout">${connTimeout:60000}</int
> > > </shardHandlerFactory
> > > </solr
> > >
> > > =======================================
> > >
> > >
> > >
> > >
> > > Any help appreciated.
> > >
> > > Regards,
> > > Piyush
> > >
> >
> >
Re: Getting Error - Session expired for /collections/sprod/state.json
Posted by Piyush Kunal <pi...@myntra.com>.
Anyone has noticed such issue before?
On Thu, Dec 15, 2016 at 4:36 PM, Piyush Kunal <pi...@myntra.com>
wrote:
> This is happening when heavy indexing like 100/second is going on.
>
> On Thu, Dec 15, 2016 at 4:33 PM, Piyush Kunal <pi...@myntra.com>
> wrote:
>
>> - We have solr6.1.0 cluster running on production with 1 shard and 5
>> replicas.
>> - Zookeeper quorum on 3 nodes.
>> - Using a chroot in zookeeper to segregate the configs from other
>> collections.
>> - Using solrj5.1.0 as our client to query solr.
>>
>>
>>
>> Usually things work fine but on and off we witness this exception coming
>> up:
>> =============================================================
>> org.apache.solr.common.SolrException: Could not load collection from
>> ZK:sprod
>> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
>> (ZkStateReader.java:815)
>> at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
>> er.java:477)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
>> ection(CloudSolrClient.java:1174)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
>> hRetryOnStaleState(CloudSolrClient.java:807)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
>> oudSolrClient.java:782)
>> --
>> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /collections/sprod/state.json
>> at org.apache.zookeeper.KeeperException.create(KeeperException.
>> java:127)
>> at org.apache.zookeeper.KeeperException.create(KeeperException.
>> java:51)
>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> ient.java:311)
>> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> ient.java:308)
>> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
>> CmdExecutor.java:61)
>> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
>> t.java:308)
>> --
>> org.apache.solr.common.SolrException: Could not load collection from
>> ZK:sprod
>> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
>> (ZkStateReader.java:815)
>> at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
>> er.java:477)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
>> ection(CloudSolrClient.java:1174)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
>> hRetryOnStaleState(CloudSolrClient.java:807)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
>> oudSolrClient.java:782)
>> --
>> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /collections/sprod/state.json
>> at org.apache.zookeeper.KeeperException.create(KeeperException.
>> java:127)
>> at org.apache.zookeeper.KeeperException.create(KeeperException.
>> java:51)
>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> ient.java:311)
>> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> ient.java:308)
>> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
>> CmdExecutor.java:61)
>> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
>> t.java:308)
>> =============================================================
>>
>>
>>
>>
>>
>> This is our zoo.cfg:
>> ======================================
>> tickTime=2000
>> dataDir=/var/lib/zookeeper
>> clientPort=2181
>> initLimit=5
>> syncLimit=2
>> server.1=192.168.70.27:2888:3888
>> server.2=192.168.70.64:2889:3889
>> server.3=192.168.70.26:2889:3889
>> maxClientCnxns=300
>> maxSessionTimeout=90000
>> =======================================
>>
>>
>>
>>
>>
>> This is our solr.xml on server side
>> =======================================
>>
>> <solr>
>>
>> <solrcloud>
>>
>> <str name="host">${host:}</str>
>> <int name="hostPort">${jetty.port:8983}</int>
>> <str name="hostContext">${hostContext:solr}</str>
>>
>> <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
>>
>> <int name="zkClientTimeout">${zkClientTimeout:30000}</int>
>> <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:600000}</int>
>> <int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:60000}</int>
>> <str name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}</str>
>> <str name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}</str>
>>
>> </solrcloud>
>>
>> <shardHandlerFactory name="shardHandlerFactory"
>> class="HttpShardHandlerFactory">
>> <int name="socketTimeout">${socketTimeout:600000}</int>
>> <int name="connTimeout">${connTimeout:60000}</int>
>> </shardHandlerFactory>
>> </solr>
>>
>> =======================================
>>
>>
>>
>>
>> Any help appreciated.
>>
>> Regards,
>> Piyush
>>
>
>
Re: Getting Error - Session expired for /collections/sprod/state.json
Posted by Piyush Kunal <pi...@myntra.com>.
This is happening when heavy indexing like 100/second is going on.
On Thu, Dec 15, 2016 at 4:33 PM, Piyush Kunal <pi...@myntra.com>
wrote:
> - We have solr6.1.0 cluster running on production with 1 shard and 5
> replicas.
> - Zookeeper quorum on 3 nodes.
> - Using a chroot in zookeeper to segregate the configs from other
> collections.
> - Using solrj5.1.0 as our client to query solr.
>
>
>
> Usually things work fine but on and off we witness this exception coming
> up:
> =============================================================
> org.apache.solr.common.SolrException: Could not load collection from
> ZK:sprod
> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(
> ZkStateReader.java:815)
> at org.apache.solr.common.cloud.ZkStateReader$5.get(
> ZkStateReader.java:477)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(
> CloudSolrClient.java:1174)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:807)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(
> CloudSolrClient.java:782)
> --
> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /collections/sprod/state.json
> at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:311)
> at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:308)
> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:61)
> at org.apache.solr.common.cloud.SolrZkClient.exists(
> SolrZkClient.java:308)
> --
> org.apache.solr.common.SolrException: Could not load collection from
> ZK:sprod
> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(
> ZkStateReader.java:815)
> at org.apache.solr.common.cloud.ZkStateReader$5.get(
> ZkStateReader.java:477)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(
> CloudSolrClient.java:1174)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:807)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(
> CloudSolrClient.java:782)
> --
> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /collections/sprod/state.json
> at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:311)
> at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:308)
> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:61)
> at org.apache.solr.common.cloud.SolrZkClient.exists(
> SolrZkClient.java:308)
> =============================================================
>
>
>
>
>
> This is our zoo.cfg:
> ======================================
> tickTime=2000
> dataDir=/var/lib/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.1=192.168.70.27:2888:3888
> server.2=192.168.70.64:2889:3889
> server.3=192.168.70.26:2889:3889
> maxClientCnxns=300
> maxSessionTimeout=90000
> =======================================
>
>
>
>
>
> This is our solr.xml on server side
> =======================================
>
> <solr>
>
> <solrcloud>
>
> <str name="host">${host:}</str>
> <int name="hostPort">${jetty.port:8983}</int>
> <str name="hostContext">${hostContext:solr}</str>
>
> <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
>
> <int name="zkClientTimeout">${zkClientTimeout:30000}</int>
> <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:600000}</int>
> <int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:60000}</int>
> <str name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}</str>
> <str name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}</str>
>
> </solrcloud>
>
> <shardHandlerFactory name="shardHandlerFactory"
> class="HttpShardHandlerFactory">
> <int name="socketTimeout">${socketTimeout:600000}</int>
> <int name="connTimeout">${connTimeout:60000}</int>
> </shardHandlerFactory>
> </solr>
>
> =======================================
>
>
>
>
> Any help appreciated.
>
> Regards,
> Piyush
>