Two problems (missing updates and timeouts)

[Aaron Jensen <aa...@gmail.com>]

Hi all, 

I'm having two problem with our solr implementation. I don't have a lot of detail about them because we're just starting to get into diagnosing them. I'm hoping for some help with that diagnosis, ideas, tips, whatever.

Our stack:

Rails
Sunspot Solr
sunspot_index_queue
two solr servers, master and slave, all traffic currently going to master, slave is just a replication slave/backup.


The first and biggest problem is that we occasionally "lose" updates. Something will get added to the database, it will trigger a solr update, but then we can't search for that thing. It's just gone. indexing that thing again will have it show up. There are a number of moving parts in our stack and this is a relatively new problem. It was working fine for 1.5 years without a problem. We're considering adding a delayed job that will index anything that is newly created a second after it is created just to "be sure" but this is a giant hack. Any ideas around this would be helpful.



The second problem is that we get occasional timeouts. These don't happen very often, maybe 5-7/day. Solr is serving at most like 350 requests per minute. Our timeouts are set to 2 seconds on read and 1 second on open. Average response time is around 20ms. It doesn't seem like any requests should be timing out but they are. I have no idea how to debug it either. Any ideas?

Thanks,

Aaron 

Re: Two problems (missing updates and timeouts)

[Erick Erickson <er...@gmail.com>]

For your first problem I'd be looking at the solr logs and verifying that
1> the update was sent
2> no stack traces are thrown
3> You probably already know all about commits, but just in case the commit
interval is passed.

For your second problem, I'm not quite sure where you're setting these
timeouts. SolrJ?

Best
Erick


On Sat, Mar 23, 2013 at 4:23 PM, Aaron Jensen <aa...@gmail.com> wrote:

> Hi all,
>
> I'm having two problem with our solr implementation. I don't have a lot of
> detail about them because we're just starting to get into diagnosing them.
> I'm hoping for some help with that diagnosis, ideas, tips, whatever.
>
> Our stack:
>
> Rails
> Sunspot Solr
> sunspot_index_queue
> two solr servers, master and slave, all traffic currently going to master,
> slave is just a replication slave/backup.
>
>
> The first and biggest problem is that we occasionally "lose" updates.
> Something will get added to the database, it will trigger a solr update,
> but then we can't search for that thing. It's just gone. indexing that
> thing again will have it show up. There are a number of moving parts in our
> stack and this is a relatively new problem. It was working fine for 1.5
> years without a problem. We're considering adding a delayed job that will
> index anything that is newly created a second after it is created just to
> "be sure" but this is a giant hack. Any ideas around this would be helpful.
>
>
>
> The second problem is that we get occasional timeouts. These don't happen
> very often, maybe 5-7/day. Solr is serving at most like 350 requests per
> minute. Our timeouts are set to 2 seconds on read and 1 second on open.
> Average response time is around 20ms. It doesn't seem like any requests
> should be timing out but they are. I have no idea how to debug it either.
> Any ideas?
>
> Thanks,
>
> Aaron
>
>