SolrCloud running away with resources

[ltenny <lt...@gmail.com>]

I've set up a simple 10 node, 5 shard SolrCloud 4.3. I'm pushing just a few
thousand documents into it.  What I'm doing is rather write intensive
100x...more writes than reads.  I've noticed that there seems to be an
unbounded use of resources.  I'm seeing a steadily increasing number of
network connections (monitored via: netstat | wc -l, which return over 5,500
and growing about 50 per minute) and over 2,200 open file descriptors (as
shown on the Solr dashboard).  This seems like there is something not
configured correctly.  At some point, rather soon I'm afraid, I'll run out
of resources.



--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-running-away-with-resources-tp4067154.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud running away with resources

[Erick Erickson <er...@gmail.com>]

Thanks for letting us know!

Erick

On Fri, May 31, 2013 at 2:42 PM, ltenny <lt...@gmail.com> wrote:
> Thanks!  I've found the cause for this problem was the hardware load balancer
> (F5 LTM) was creating thousands of connections.  So it turns out that it had
> nothing to do with SolrCloud.  However, now I have another problem for which
> I'll create another post.
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-running-away-with-resources-tp4067154p4067426.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud running away with resources

[ltenny <lt...@gmail.com>]

Thanks!  I've found the cause for this problem was the hardware load balancer
(F5 LTM) was creating thousands of connections.  So it turns out that it had
nothing to do with SolrCloud.  However, now I have another problem for which
I'll create another post.





--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-running-away-with-resources-tp4067154p4067426.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud running away with resources

[Erick Erickson <er...@gmail.com>]

Shouldn't be happening, although a lot of SolrCloud is new.....

So, what are your autocommit intervals, both hard and soft?
That's the place where things go weird fast. Oddly, consider
making your hard commit interval relatively short (say 10000)
with openSearcher=false. This is just a check, it doesn't sound
like your root problem, but that will truncate your transaction logs.

What is your ingestion rate? Do you know whether your
system is batching up lots and lots and lots of docs? Or is
your cluster keeping up with your indexing speed?

As you can tell, I don't have a pat answer, but something sure
seems out of whack or you've uncovered an edge case....

Best
Erick

On Thu, May 30, 2013 at 3:40 PM, ltenny <lt...@gmail.com> wrote:
> I've set up a simple 10 node, 5 shard SolrCloud 4.3. I'm pushing just a few
> thousand documents into it.  What I'm doing is rather write intensive
> 100x...more writes than reads.  I've noticed that there seems to be an
> unbounded use of resources.  I'm seeing a steadily increasing number of
> network connections (monitored via: netstat | wc -l, which return over 5,500
> and growing about 50 per minute) and over 2,200 open file descriptors (as
> shown on the Solr dashboard).  This seems like there is something not
> configured correctly.  At some point, rather soon I'm afraid, I'll run out
> of resources.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-running-away-with-resources-tp4067154.html
> Sent from the Solr - User mailing list archive at Nabble.com.