You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by neoman <ha...@gmail.com> on 2013/09/17 17:40:34 UTC
Solr node goes down while trying to index records
Hello everyone,
one or more of the nodes in the solrcloud go down randomly when we try to
index data using solrj APIs. The nodes do recover. but when we try to index
back, they go down again
Our configuration:
3 shards
Solr 4.4.
I see the following exceptions in the log file.
<09/17/13
15:33:32:976|localhost-startStop-1-SendThread(10.68.129.119:9080)|INFO|org.apache.zookeeper.ClientCnxn|Socket
connection established to 10.68.129.119/10.68.129.119:9080, initiating
session|
<09/17/13
15:33:32:978|localhost-startStop-1-SendThread(10.68.129.119:9080)|INFO|org.apache.zookeeper.ClientCnxn|Unable
to reconnect to ZooKeeper service, session 0x34109f9474b0029 has expired,
closing socket connection|
<09/17/13
15:34:36:080|localhost-startStop-1-EventThread|ERROR|apache.solr.cloud.ZkController|There
was a problem making a request to the
leader:org.apache.solr.client.solrj.SolrServerException: Timeout occured
while waiting response from server at: http://solr02-prod.phneaz:8080/solr
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:431)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at
org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1421)
at
org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:306)
at
org.apache.solr.cloud.ZkController.access$100(ZkController.java:86)
at
org.apache.solr.cloud.ZkController$1.command(ZkController.java:196)
at
org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:117)
at
org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:46)
at
org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:91)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
We are also getting IOExcpetion in the client side.
Adding chunk 122
Total Count 12422
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr-prod.com:8443/solr/aq-collection
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:409)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
at
com.billmelater.fraudworkstation.data.DataProvider.flushBatch(DataProvider.java:48)
at
com.billmelater.fraudworkstation.data.AQDBDataProvider.execute(AQDBDataProvider.java:114)
at
com.billmelater.fraudworkstation.data.AQDBDataProvider.main(AQDBDataProvider.java:244)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
Your help is appreciated.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-node-goes-down-while-trying-to-index-records-tp4090610.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr node goes down while trying to index records
Posted by Erick Erickson <er...@gmail.com>.
What happens if you bump up you zookeeper timeout? This has been an
issue at times in the past.
Best,
Erick
On Tue, Sep 17, 2013 at 1:48 PM, Furkan KAMACI <fu...@gmail.com> wrote:
> Could you give some information about your jetty.xml and give more info
> about your index rate and RAM usage of your machines?
>
> 17 Eylül 2013 Salı tarihinde neoman <ha...@gmail.com> adlı kullanıcı
> şöyle yazdı:
>> yes. the nodes go down while indexing. if we stop indexing, it does not go
>> down.
>>
>>
>>
>> --
>> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-node-goes-down-while-trying-to-index-records-tp4090610p4090644.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
Re: Solr node goes down while trying to index records
Posted by Erick Erickson <er...@gmail.com>.
What happens if you bump up you zookeeper timeout? This has been an issue
at times in the past.
Best,
Erick
On Tue, Sep 17, 2013 at 1:48 PM, Furkan KAMACI <fu...@gmail.com>wrote:
> Could you give some information about your jetty.xml and give more info
> about your index rate and RAM usage of your machines?
>
> 17 Eylül 2013 Salı tarihinde neoman <ha...@gmail.com> adlı kullanıcı
> şöyle yazdı:
> > yes. the nodes go down while indexing. if we stop indexing, it does not
> go
> > down.
> >
> >
> >
> > --
> > View this message in context:
>
> http://lucene.472066.n3.nabble.com/Solr-node-goes-down-while-trying-to-index-records-tp4090610p4090644.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
Re: Solr node goes down while trying to index records
Posted by Furkan KAMACI <fu...@gmail.com>.
Could you give some information about your jetty.xml and give more info
about your index rate and RAM usage of your machines?
17 Eylül 2013 Salı tarihinde neoman <ha...@gmail.com> adlı kullanıcı
şöyle yazdı:
> yes. the nodes go down while indexing. if we stop indexing, it does not go
> down.
>
>
>
> --
> View this message in context:
http://lucene.472066.n3.nabble.com/Solr-node-goes-down-while-trying-to-index-records-tp4090610p4090644.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr node goes down while trying to index records
Posted by neoman <ha...@gmail.com>.
yes. the nodes go down while indexing. if we stop indexing, it does not go
down.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-node-goes-down-while-trying-to-index-records-tp4090610p4090644.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr node goes down while trying to index records
Posted by Furkan KAMACI <fu...@gmail.com>.
Do you get that error only when indexing?
2013/9/17 neoman <ha...@gmail.com>
> Hello everyone,
> one or more of the nodes in the solrcloud go down randomly when we try to
> index data using solrj APIs. The nodes do recover. but when we try to index
> back, they go down again
>
> Our configuration:
> 3 shards
> Solr 4.4.
>
> I see the following exceptions in the log file.
> <09/17/13
> 15:33:32:976|localhost-startStop-1-SendThread(10.68.129.119:9080
> )|INFO|org.apache.zookeeper.ClientCnxn|Socket
> connection established to 10.68.129.119/10.68.129.119:9080, initiating
> session|
> <09/17/13
> 15:33:32:978|localhost-startStop-1-SendThread(10.68.129.119:9080
> )|INFO|org.apache.zookeeper.ClientCnxn|Unable
> to reconnect to ZooKeeper service, session 0x34109f9474b0029 has expired,
> closing socket connection|
> <09/17/13
>
> 15:34:36:080|localhost-startStop-1-EventThread|ERROR|apache.solr.cloud.ZkController|There
> was a problem making a request to the
> leader:org.apache.solr.client.solrj.SolrServerException: Timeout occured
> while waiting response from server at: http://solr02-prod.phneaz:8080/solr
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:431)
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
> at
>
> org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1421)
> at
>
> org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:306)
> at
> org.apache.solr.cloud.ZkController.access$100(ZkController.java:86)
> at
> org.apache.solr.cloud.ZkController$1.command(ZkController.java:196)
> at
>
> org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:117)
> at
>
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:46)
> at
>
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:91)
> at
>
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
>
> We are also getting IOExcpetion in the client side.
> Adding chunk 122
> Total Count 12422
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://solr-prod.com:8443/solr/aq-collection
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:409)
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
> at
>
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
> at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
> at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
> at
>
> com.billmelater.fraudworkstation.data.DataProvider.flushBatch(DataProvider.java:48)
> at
>
> com.billmelater.fraudworkstation.data.AQDBDataProvider.execute(AQDBDataProvider.java:114)
> at
>
> com.billmelater.fraudworkstation.data.AQDBDataProvider.main(AQDBDataProvider.java:244)
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>
> Your help is appreciated.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-node-goes-down-while-trying-to-index-records-tp4090610.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>