SolrCloud - Service unavailable

[Indika Tantrigoda <in...@gmail.com>]

Hi All,

I am using 3 Solr instances behind an Amazon ELB with 1 shared. Serving
data via Solr works as expected, however I noticed a few times a 503 error
was poping up from the applications accessing Solr. Accessing Solr is done
via the AWS ELB.

3 Zookeeper instances also run on the same instances as Solr on a separate
disk.

Solr version is 4.4.

This issue seems to be a sporadic issue. Has anyone else observed this kind
of behavior ?

Thanks,
Indika

Re: SolrCloud - Service unavailable

[Indika Tantrigoda <in...@gmail.com>]

Thanks Shawn, the links will be useful.

I am still not sure if its related due to a timeout because the 503 error
is coming from Tomcat, which means the requests are going through. I can
access the Solr admin panel and I see a message saying the core was not
initialized.

Thanks,
Indika


On 18 September 2013 21:27, Shawn Heisey <so...@elyograg.org> wrote:

> On 9/18/2013 8:12 AM, Indika Tantrigoda wrote:
> > I am using 3 Solr instances behind an Amazon ELB with 1 shared. Serving
> > data via Solr works as expected, however I noticed a few times a 503
> error
> > was poping up from the applications accessing Solr. Accessing Solr is
> done
> > via the AWS ELB.
> >
> > 3 Zookeeper instances also run on the same instances as Solr on a
> separate
> > disk.
> >
> > Solr version is 4.4.
> >
> > This issue seems to be a sporadic issue. Has anyone else observed this
> kind
> > of behavior ?
>
> What kind of session timeouts have you configured on the amazon load
> balancer?  I've never used amazon services, but hopefully this is
> configurable.  If the timeout is low enough, it could be just that the
> request is taking longer than that to execute.  You may need to increase
> that timeout.
>
> Aside from general performance issues, one thing that can cause long
> request times is stop-the-world Java garbage collections.  This can be a
> sign that your heap is too small, too large, or that your garbage
> collection hasn't been properly tuned.
>
> http://wiki.apache.org/solr/SolrPerformanceProblems#GC_pause_problems
>
> http://wiki.apache.org/solr/SolrPerformanceProblems#How_much_heap_space_do_I_need.3F
>
> That same wiki page has another section about the OS disk cache.  Not
> having enough memory for this is the cause of a lot of performance issues:
>
> http://wiki.apache.org/solr/SolrPerformanceProblems#OS_Disk_Cache
>
> Thanks,
> Shawn
>
>

Re: SolrCloud - Service unavailable

[Shawn Heisey <so...@elyograg.org>]

On 9/18/2013 8:12 AM, Indika Tantrigoda wrote:
> I am using 3 Solr instances behind an Amazon ELB with 1 shared. Serving
> data via Solr works as expected, however I noticed a few times a 503 error
> was poping up from the applications accessing Solr. Accessing Solr is done
> via the AWS ELB.
> 
> 3 Zookeeper instances also run on the same instances as Solr on a separate
> disk.
> 
> Solr version is 4.4.
> 
> This issue seems to be a sporadic issue. Has anyone else observed this kind
> of behavior ?

What kind of session timeouts have you configured on the amazon load
balancer?  I've never used amazon services, but hopefully this is
configurable.  If the timeout is low enough, it could be just that the
request is taking longer than that to execute.  You may need to increase
that timeout.

Aside from general performance issues, one thing that can cause long
request times is stop-the-world Java garbage collections.  This can be a
sign that your heap is too small, too large, or that your garbage
collection hasn't been properly tuned.

http://wiki.apache.org/solr/SolrPerformanceProblems#GC_pause_problems
http://wiki.apache.org/solr/SolrPerformanceProblems#How_much_heap_space_do_I_need.3F

That same wiki page has another section about the OS disk cache.  Not
having enough memory for this is the cause of a lot of performance issues:

http://wiki.apache.org/solr/SolrPerformanceProblems#OS_Disk_Cache

Thanks,
Shawn