You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Kent Mu <so...@gmail.com> on 2016/07/06 11:26:38 UTC
sorlcloud connection issue
Hi friends!
*solr version: 4.9.0*
I came across a problem when use solrcloud, it becomes dead lock, we got
the java core log, it looks like the http connection pool is exhausted and
most threads are waiting to get a free connection..
I posted the problem in JIRA, the link is
https://issues.apache.org/jira/browse/SOLR-9253
I have increased http connection defaults for the SolrJ client, and also
configed the connection defaults in solr.xml for all shard servers as below.
<shardHandlerFactory name="shardHandlerFactory"
class="HttpShardHandlerFactory">
<int name="socketTimeout">60000</int>
<int name="connTimeout">30000</int>
<int name="maxConnections">10000</int>
<int name="maxConnectionsPerHost">500</int>
</shardHandlerFactory>
*besides, we use the singleton pattern to connect solrlcoud.*
public synchronized static CloudSolrServer getSolrCloudReadServer() {
if (reviewSolrCloudServer == null) {
ModifiableSolrParams params = new ModifiableSolrParams();
params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 2000);
params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 500);
HttpClient client = HttpClientUtil.createClient(params);
LBHttpSolrServer lbServer = new LBHttpSolrServer(client);
lbServer.setConnectionTimeout(ReviewProperties.getCloudConnectionTimeOut());
lbServer.setSoTimeout(ReviewProperties.getCloudSoTimeOut());
reviewSolrCloudServer = new
CloudSolrServer(ReviewProperties.getZkHost(),lbServer);
reviewSolrCloudServer.setDefaultCollection("commodityReview");
reviewSolrCloudServer.setZkClientTimeout(ReviewProperties.getZkClientTimeout());
reviewSolrCloudServer.setZkConnectTimeout(ReviewProperties.getZkConnectTimeout());
reviewSolrCloudServer.connect();
}
return reviewSolrCloudServer;
}
*the java stack as below*
"httpShardExecutor-3-thread-541" prio=10 tid=0x00007f7b1c02b000 nid=0x20af
waiting on condition [0x00007f79fd490000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000605710068> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133)
at
org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282)
at
org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
at
org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177)
at
org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170)
at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102)
at
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:208)
at
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
at
org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
at
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
at
org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:205)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
please help me, look forward to your reply. Thanks a lot!
Best Regards!
Kent
Re: sorlcloud connection issue
Posted by Kent Mu <so...@gmail.com>.
Dear Mr. Mangar!
Thanks for your reply. we have 3 cores with 5 shards, and each shard with
one leader and one replication. now the data account goes to be 100
million, and it is still growing up.I see no connection timeout. actually
we have schedule job to commit the data every 2 hours. so the action to
commit is not that frequent, and no auto-commit. the write operation seems
to be OK.
I just see the reply by Mr. Heisey. He finds "jboss running info in java
core". and he says that we need to raise the maxThread in JBOSS, which is
200 as default. so I decide to config it on next week, and to see if it
will be OK.
Thanks again!
2016-07-07 16:08 GMT+08:00 Shalin Shekhar Mangar <sh...@gmail.com>:
> Hi Kent,
>
> There is no point sending multiple emails for the same subject. It
> distracts people from the other messages, distributes the conversation and
> discourages people from helping you.
>
> Please provide more details about your cluster.
> 1. How many nodes?
> 2. How many collections?
> 3. How many shards?
> 4. What is the replication factor?
> 5. Are there connection and read timeouts specified on both client and
> server? If yes, what are the values?
> 6. How often do you commit? Is there auto-commit or explicit commits from
> clients?
> 7. What else is happening on your cluster? What is the write and query
> load?
>
> The important thing is to figure out why these pools are getting exhausted.
> 10000 max connections and 500 connections per host is already pretty high
> so there is an underlying cause for such stuck requests.
>
> On Wed, Jul 6, 2016 at 4:56 PM, Kent Mu <so...@gmail.com> wrote:
>
> > Hi friends!
> > *solr version: 4.9.0*
> >
> > I came across a problem when use solrcloud, it becomes dead lock, we got
> > the java core log, it looks like the http connection pool is exhausted
> and
> > most threads are waiting to get a free connection..
> >
> > I posted the problem in JIRA, the link is
> > https://issues.apache.org/jira/browse/SOLR-9253
> > I have increased http connection defaults for the SolrJ client, and also
> > configed the connection defaults in solr.xml for all shard servers as
> > below.
> >
> > <shardHandlerFactory name="shardHandlerFactory"
> > class="HttpShardHandlerFactory">
> > <int name="socketTimeout">60000</int>
> > <int name="connTimeout">30000</int>
> > <int name="maxConnections">10000</int>
> > <int name="maxConnectionsPerHost">500</int>
> > </shardHandlerFactory>
> >
> > *besides, we use the singleton pattern to connect solrlcoud.*
> >
> > public synchronized static CloudSolrServer getSolrCloudReadServer() {
> > if (reviewSolrCloudServer == null) {
> > ModifiableSolrParams params = new ModifiableSolrParams();
> > params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 2000);
> > params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 500);
> > HttpClient client = HttpClientUtil.createClient(params);
> > LBHttpSolrServer lbServer = new LBHttpSolrServer(client);
> >
> >
> >
> lbServer.setConnectionTimeout(ReviewProperties.getCloudConnectionTimeOut());
> > lbServer.setSoTimeout(ReviewProperties.getCloudSoTimeOut());
> >
> > reviewSolrCloudServer = new
> > CloudSolrServer(ReviewProperties.getZkHost(),lbServer);
> > reviewSolrCloudServer.setDefaultCollection("commodityReview");
> >
> >
> >
> reviewSolrCloudServer.setZkClientTimeout(ReviewProperties.getZkClientTimeout());
> >
> >
> >
> reviewSolrCloudServer.setZkConnectTimeout(ReviewProperties.getZkConnectTimeout());
> > reviewSolrCloudServer.connect();
> > }
> > return reviewSolrCloudServer;
> > }
> >
> > *the java stack as below*
> >
> > "httpShardExecutor-3-thread-541" prio=10 tid=0x00007f7b1c02b000
> nid=0x20af
> > waiting on condition [0x00007f79fd490000]
> > java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <0x0000000605710068> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> > at
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> > at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133)
> > at
> >
> >
> org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282)
> > at
> >
> org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
> > at
> >
> >
> org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177)
> > at
> >
> >
> org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170)
> > at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102)
> > at
> >
> >
> org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:208)
> > at
> >
> >
> org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)
> > at
> >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422)
> > at
> >
> >
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
> > at
> >
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> > at
> >
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
> > at
> >
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
> > at
> >
> >
> org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:205)
> > at
> >
> >
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162)
> > at
> >
> >
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:724)
> >
> > please help me, look forward to your reply. Thanks a lot!
> >
> > Best Regards!
> > Kent
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: sorlcloud connection issue
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Hi Kent,
There is no point sending multiple emails for the same subject. It
distracts people from the other messages, distributes the conversation and
discourages people from helping you.
Please provide more details about your cluster.
1. How many nodes?
2. How many collections?
3. How many shards?
4. What is the replication factor?
5. Are there connection and read timeouts specified on both client and
server? If yes, what are the values?
6. How often do you commit? Is there auto-commit or explicit commits from
clients?
7. What else is happening on your cluster? What is the write and query load?
The important thing is to figure out why these pools are getting exhausted.
10000 max connections and 500 connections per host is already pretty high
so there is an underlying cause for such stuck requests.
On Wed, Jul 6, 2016 at 4:56 PM, Kent Mu <so...@gmail.com> wrote:
> Hi friends!
> *solr version: 4.9.0*
>
> I came across a problem when use solrcloud, it becomes dead lock, we got
> the java core log, it looks like the http connection pool is exhausted and
> most threads are waiting to get a free connection..
>
> I posted the problem in JIRA, the link is
> https://issues.apache.org/jira/browse/SOLR-9253
> I have increased http connection defaults for the SolrJ client, and also
> configed the connection defaults in solr.xml for all shard servers as
> below.
>
> <shardHandlerFactory name="shardHandlerFactory"
> class="HttpShardHandlerFactory">
> <int name="socketTimeout">60000</int>
> <int name="connTimeout">30000</int>
> <int name="maxConnections">10000</int>
> <int name="maxConnectionsPerHost">500</int>
> </shardHandlerFactory>
>
> *besides, we use the singleton pattern to connect solrlcoud.*
>
> public synchronized static CloudSolrServer getSolrCloudReadServer() {
> if (reviewSolrCloudServer == null) {
> ModifiableSolrParams params = new ModifiableSolrParams();
> params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 2000);
> params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 500);
> HttpClient client = HttpClientUtil.createClient(params);
> LBHttpSolrServer lbServer = new LBHttpSolrServer(client);
>
>
> lbServer.setConnectionTimeout(ReviewProperties.getCloudConnectionTimeOut());
> lbServer.setSoTimeout(ReviewProperties.getCloudSoTimeOut());
>
> reviewSolrCloudServer = new
> CloudSolrServer(ReviewProperties.getZkHost(),lbServer);
> reviewSolrCloudServer.setDefaultCollection("commodityReview");
>
>
> reviewSolrCloudServer.setZkClientTimeout(ReviewProperties.getZkClientTimeout());
>
>
> reviewSolrCloudServer.setZkConnectTimeout(ReviewProperties.getZkConnectTimeout());
> reviewSolrCloudServer.connect();
> }
> return reviewSolrCloudServer;
> }
>
> *the java stack as below*
>
> "httpShardExecutor-3-thread-541" prio=10 tid=0x00007f7b1c02b000 nid=0x20af
> waiting on condition [0x00007f79fd490000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000605710068> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133)
> at
>
> org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282)
> at
> org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
> at
>
> org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177)
> at
>
> org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170)
> at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102)
> at
>
> org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:208)
> at
>
> org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)
> at
>
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422)
> at
>
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
> at
>
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
>
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
> at
>
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452)
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
> at
>
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
> at
>
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
> at
>
> org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:205)
> at
>
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162)
> at
>
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
>
> please help me, look forward to your reply. Thanks a lot!
>
> Best Regards!
> Kent
>
--
Regards,
Shalin Shekhar Mangar.
Re: sorlcloud connection issue
Posted by Kent Mu <so...@gmail.com>.
we have 5 shards and each shard with one leader and one replica.
the "3300" connections is only for one JVM. please see the follow analysis
in zabbix.
and we the solrj code as follow:
public synchronized static CloudSolrServer getSolrCloudReadServer() {
if (reviewSolrCloudReadServer == null) {
ModifiableSolrParams params = new ModifiableSolrParams();
params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 1000);
params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 100);
HttpClient client = HttpClientUtil.createClient(params);
LBHttpSolrServer lbServer = new LBHttpSolrServer(client);
lbServer.setConnectionTimeout(ReviewProperties.getCloudConnectionTimeOut());
lbServer.setSoTimeout(ReviewProperties.getCloudSoTimeOut());
reviewSolrCloudReadServer = new
CloudSolrServer(ReviewProperties.getZkHost(),lbServer);
reviewSolrCloudReadServer.setDefaultCollection(ReviewProperties.getZkReviewConnection());
reviewSolrCloudReadServer.setZkClientTimeout(ReviewProperties.getZkClientTimeout());
reviewSolrCloudReadServer.setZkConnectTimeout(ReviewProperties.getZkConnectTimeout());
}
return reviewSolrCloudReadServer;
}
2016-07-13 8:13 GMT+08:00 Shawn Heisey <ap...@elyograg.org>:
> On 7/12/2016 8:30 AM, Kent Mu wrote:
> > We have configed the maxThreads in JBOSS, and the good news is solrcloud
> > now running OK. but I another issue came across. We find the number of
> the
> > HTTP connections is very high, and the number can be around 3300. and
> > solrcloud does no release the connections.
> > I understand that, the solrcloud needs to connect to zookeeper and
> > communication between leader and replica need the connection. but I think
> > the number should not to be so huge.
> > besides, we use the singleton pattern to connect solrcloud in JAVA.
>
> Are you referring to the number of http connections in your SolrJ app,
> or the number of http connections in Solr itself? Hopefully these are
> being run by completely separate JVMs. Where exactly are you looking
> when you see 3300 connections?
>
> The connection to Zookeeper does not use HTTP. It is a TCP connection
> but the protocol is custom. Both Solr and SolrJ will maintain a
> connection to each of the zookeeper hosts that are in the zkHost string
> used when they start.
>
> Thanks,
> Shawn
>
>
Re: sorlcloud connection issue
Posted by Kent Mu <so...@gmail.com>.
Dear Mr. Heisey.
It seems that we can not send the picture or attachments to solr-user, so I
send the screen shot to your personal email, sorry to disturb!
Thanks!
Kent
2016-07-13 8:13 GMT+08:00 Shawn Heisey <ap...@elyograg.org>:
> On 7/12/2016 8:30 AM, Kent Mu wrote:
> > We have configed the maxThreads in JBOSS, and the good news is solrcloud
> > now running OK. but I another issue came across. We find the number of
> the
> > HTTP connections is very high, and the number can be around 3300. and
> > solrcloud does no release the connections.
> > I understand that, the solrcloud needs to connect to zookeeper and
> > communication between leader and replica need the connection. but I think
> > the number should not to be so huge.
> > besides, we use the singleton pattern to connect solrcloud in JAVA.
>
> Are you referring to the number of http connections in your SolrJ app,
> or the number of http connections in Solr itself? Hopefully these are
> being run by completely separate JVMs. Where exactly are you looking
> when you see 3300 connections?
>
> The connection to Zookeeper does not use HTTP. It is a TCP connection
> but the protocol is custom. Both Solr and SolrJ will maintain a
> connection to each of the zookeeper hosts that are in the zkHost string
> used when they start.
>
> Thanks,
> Shawn
>
>
Re: sorlcloud connection issue
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/12/2016 8:30 AM, Kent Mu wrote:
> We have configed the maxThreads in JBOSS, and the good news is solrcloud
> now running OK. but I another issue came across. We find the number of the
> HTTP connections is very high, and the number can be around 3300. and
> solrcloud does no release the connections.
> I understand that, the solrcloud needs to connect to zookeeper and
> communication between leader and replica need the connection. but I think
> the number should not to be so huge.
> besides, we use the singleton pattern to connect solrcloud in JAVA.
Are you referring to the number of http connections in your SolrJ app,
or the number of http connections in Solr itself? Hopefully these are
being run by completely separate JVMs. Where exactly are you looking
when you see 3300 connections?
The connection to Zookeeper does not use HTTP. It is a TCP connection
but the protocol is custom. Both Solr and SolrJ will maintain a
connection to each of the zookeeper hosts that are in the zkHost string
used when they start.
Thanks,
Shawn
Re: sorlcloud connection issue
Posted by Kent Mu <so...@gmail.com>.
Dear Mr. Heisey.
We have configed the maxThreads in JBOSS, and the good news is solrcloud
now running OK. but I another issue came across. We find the number of the
HTTP connections is very high, and the number can be around 3300. and
solrcloud does no release the connections.
I understand that, the solrcloud needs to connect to zookeeper and
communication between leader and replica need the connection. but I think
the number should not to be so huge.
besides, we use the singleton pattern to connect solrcloud in JAVA.
look forward to your reply. Thanks!
2016-07-07 22:24 GMT+08:00 Shawn Heisey <ap...@elyograg.org>:
> On 7/6/2016 5:26 AM, Kent Mu wrote:
> > Hi friends!
> > *solr version: 4.9.0*
> >
> > I came across a problem when use solrcloud, it becomes dead lock, we got
> > the java core log, it looks like the http connection pool is exhausted
> and
> > most threads are waiting to get a free connection..
> >
> > I posted the problem in JIRA, the link is
> > https://issues.apache.org/jira/browse/SOLR-9253
> > I have increased http connection defaults for the SolrJ client, and also
> > configed the connection defaults in solr.xml for all shard servers as
> below.
> >
> > <shardHandlerFactory name="shardHandlerFactory"
> > class="HttpShardHandlerFactory">
> > <int name="socketTimeout">60000</int>
> > <int name="connTimeout">30000</int>
> > <int name="maxConnections">10000</int>
> > <int name="maxConnectionsPerHost">500</int>
> > </shardHandlerFactory>
>
> I can see JBoss classes in the thread dump that was added to SOLR-9253.
>
> That thread dump shows 213 threads in the RUNNABLE state, and 507 in the
> WAITING state. I do not think you are running into the configured shard
> handler limits. I think your container is not allowing enough Solr
> threads to run.
>
> Just like Tomcat and Jetty, JBoss has a "maxThreads" setting that
> defaults to 200. Increasing this setting is critical for scalability
> when using a third-party container. I recommend 10000 -- which is the
> setting you'll find in the Jetty that's included with Solr.
>
> Note that if you upgrade Solr to 5.x or 6.x, running in JBoss will no
> longer be a supported configuration.
>
> https://wiki.apache.org/solr/WhyNoWar
>
> Thanks,
> Shawn
>
>
Re: sorlcloud connection issue
Posted by Kent Mu <so...@gmail.com>.
Dear Mr. Heisey!
Thanks for your reply very much! as you said, I think this will be key to
the issue, and I will config "maxThreads" next week. hope it will work.
Thanks again!
2016-07-07 22:24 GMT+08:00 Shawn Heisey <ap...@elyograg.org>:
> On 7/6/2016 5:26 AM, Kent Mu wrote:
> > Hi friends!
> > *solr version: 4.9.0*
> >
> > I came across a problem when use solrcloud, it becomes dead lock, we got
> > the java core log, it looks like the http connection pool is exhausted
> and
> > most threads are waiting to get a free connection..
> >
> > I posted the problem in JIRA, the link is
> > https://issues.apache.org/jira/browse/SOLR-9253
> > I have increased http connection defaults for the SolrJ client, and also
> > configed the connection defaults in solr.xml for all shard servers as
> below.
> >
> > <shardHandlerFactory name="shardHandlerFactory"
> > class="HttpShardHandlerFactory">
> > <int name="socketTimeout">60000</int>
> > <int name="connTimeout">30000</int>
> > <int name="maxConnections">10000</int>
> > <int name="maxConnectionsPerHost">500</int>
> > </shardHandlerFactory>
>
> I can see JBoss classes in the thread dump that was added to SOLR-9253.
>
> That thread dump shows 213 threads in the RUNNABLE state, and 507 in the
> WAITING state. I do not think you are running into the configured shard
> handler limits. I think your container is not allowing enough Solr
> threads to run.
>
> Just like Tomcat and Jetty, JBoss has a "maxThreads" setting that
> defaults to 200. Increasing this setting is critical for scalability
> when using a third-party container. I recommend 10000 -- which is the
> setting you'll find in the Jetty that's included with Solr.
>
> Note that if you upgrade Solr to 5.x or 6.x, running in JBoss will no
> longer be a supported configuration.
>
> https://wiki.apache.org/solr/WhyNoWar
>
> Thanks,
> Shawn
>
>
Re: sorlcloud connection issue
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/6/2016 5:26 AM, Kent Mu wrote:
> Hi friends!
> *solr version: 4.9.0*
>
> I came across a problem when use solrcloud, it becomes dead lock, we got
> the java core log, it looks like the http connection pool is exhausted and
> most threads are waiting to get a free connection..
>
> I posted the problem in JIRA, the link is
> https://issues.apache.org/jira/browse/SOLR-9253
> I have increased http connection defaults for the SolrJ client, and also
> configed the connection defaults in solr.xml for all shard servers as below.
>
> <shardHandlerFactory name="shardHandlerFactory"
> class="HttpShardHandlerFactory">
> <int name="socketTimeout">60000</int>
> <int name="connTimeout">30000</int>
> <int name="maxConnections">10000</int>
> <int name="maxConnectionsPerHost">500</int>
> </shardHandlerFactory>
I can see JBoss classes in the thread dump that was added to SOLR-9253.
That thread dump shows 213 threads in the RUNNABLE state, and 507 in the
WAITING state. I do not think you are running into the configured shard
handler limits. I think your container is not allowing enough Solr
threads to run.
Just like Tomcat and Jetty, JBoss has a "maxThreads" setting that
defaults to 200. Increasing this setting is critical for scalability
when using a third-party container. I recommend 10000 -- which is the
setting you'll find in the Jetty that's included with Solr.
Note that if you upgrade Solr to 5.x or 6.x, running in JBoss will no
longer be a supported configuration.
https://wiki.apache.org/solr/WhyNoWar
Thanks,
Shawn