New docs need server restart after synchronization

[galo <ga...@last.fm>]

Hi there,

I've been running an index on 3 nodes (master + 2 slaves) without 
problems for a few weeks, snappuller and snapinstaller run every hour 
normally, install the new index without errors etc.

In the last couple of days it seems like I need to restart tomcat for 
the new documents to appear in the slave indexes. New docs are updated, 
commited and appear if I do a search on the master, but if after the 
synchronization I search in the slaves they don't appear. If I restart 
tomcat they do.

I remember this happening while I was setting up the index and seeing an 
error somewhere (something like it not being able to open a new searcher 
if i remember well), but i'm pretty sure the configuration is ok, it's 
been working for weeks normally and I haven't done any changes since then..

any ideas where i should look at? I can't see any error messages in the 
tomcat logs or the scripts' !

thanks,
galo

Re: New docs need server restart after synchronization

[galo <ga...@last.fm>]

Yep, all normal..

galo

Bill Au wrote:
> Did you check for error messages in the snappuller and snapinstaller
> log files sunder solr/logs?  Distribution related errors will not show
> up in the tomcat logs.
> 
> Bill
> 
> On 4/16/07, galo <ga...@last.fm> wrote:
>> Hi there,
>>
>> I've been running an index on 3 nodes (master + 2 slaves) without
>> problems for a few weeks, snappuller and snapinstaller run every hour
>> normally, install the new index without errors etc.
>>
>> In the last couple of days it seems like I need to restart tomcat for
>> the new documents to appear in the slave indexes. New docs are updated,
>> commited and appear if I do a search on the master, but if after the
>> synchronization I search in the slaves they don't appear. If I restart
>> tomcat they do.
>>
>> I remember this happening while I was setting up the index and seeing an
>> error somewhere (something like it not being able to open a new searcher
>> if i remember well), but i'm pretty sure the configuration is ok, it's
>> been working for weeks normally and I haven't done any changes since 
>> then..
>>
>> any ideas where i should look at? I can't see any error messages in the
>> tomcat logs or the scripts' !
>>
>> thanks,
>> galo
>>
> 


-- 
Galo Navarro, Developer

galo@last.fm
t. +44 (0)20 7780 7080

Last.fm | http://www.last.fm
Karen House 1-11 Baches Street
London N1 6DL

http://www.last.fm/user/galeote

Re: New docs need server restart after synchronization

[Bill Au <bi...@gmail.com>]

Did you check for error messages in the snappuller and snapinstaller
log files sunder solr/logs?  Distribution related errors will not show
up in the tomcat logs.

Bill

On 4/16/07, galo <ga...@last.fm> wrote:
> Hi there,
>
> I've been running an index on 3 nodes (master + 2 slaves) without
> problems for a few weeks, snappuller and snapinstaller run every hour
> normally, install the new index without errors etc.
>
> In the last couple of days it seems like I need to restart tomcat for
> the new documents to appear in the slave indexes. New docs are updated,
> commited and appear if I do a search on the master, but if after the
> synchronization I search in the slaves they don't appear. If I restart
> tomcat they do.
>
> I remember this happening while I was setting up the index and seeing an
> error somewhere (something like it not being able to open a new searcher
> if i remember well), but i'm pretty sure the configuration is ok, it's
> been working for weeks normally and I haven't done any changes since then..
>
> any ideas where i should look at? I can't see any error messages in the
> tomcat logs or the scripts' !
>
> thanks,
> galo
>

Re: New docs need server restart after synchronization

[Chris Hostetter <ho...@fucit.org>]

: logged correctly... on the master. I had copied scripts.conf from the
: master to all the slaves with solr_hostname=master and didn't replace
: with slave1 slave2 etc, so the commit was successful, but not on the
: slave. Dumb. Dumb.

ah ... glad we found the problem.

: Now that we are at it.. how's the best set up for multiple indexes on
: the same server? Atm I have a tomcat server runs on each machine with a
: separate webapp for each index for a few indexes (5 by now, will be many
: more)

personally? .. i run a seperate servlet container instance for every index
... but that's becuase i have the RAM to spare and it helps ensure things
that each index runs in heavy isolation -- a serious flaw in one index
might crash the port, or a bad config might cause OOMs but then only one
index goes down, the other are still running.

but it's really a question of what you are comfortable with, and how much
money you have to spend.




-Hoss

Re: New docs need server restart after synchronization

[galo <ga...@last.fm>]

I was expecting to see the commit error which is the one that told me 
before the configuration was not correct but the commit was made and 
logged correctly... on the master. I had copied scripts.conf from the 
master to all the slaves with solr_hostname=master and didn't replace 
with slave1 slave2 etc, so the commit was successful, but not on the 
slave. Dumb. Dumb.

Now that we are at it.. how's the best set up for multiple indexes on 
the same server? Atm I have a tomcat server runs on each machine with a 
separate webapp for each index for a few indexes (5 by now, will be many 
more)

Something like
  master: http://master:8080/index1, http://master:8080/index2
  slave1: http://slave1:8080/index1, http://slave1:8080/index2
  slave2: http://slave2:8080/index1, http://slave2:8080/index2
  slave3: http://slave3:8080/index1, http://slave3:8080/index2
  ...

On each server I have a main solr directory and then one for each index

solr
solr-index1
solr-index2

The main directory (solr) holds all the shared folders, so inside 
solr-index1 and solr-index2, bin, etc, lib, ext, webapp etc are links to 
sorl/bin, solr/etc, solr/lib, etc. Obviously, conf, data and logs are 
not shared. This is reasonably simple to set up and install on other 
servers being the only real complain that i'd like to keep a single conf 
folder shared across master and slaves for each index (schema.xml, 
solrconfig.xml etc. should be the same and scripts.conf can be shared as 
long as you use solr_hostname=localhost and the same port, which is my 
case). Not a tragedy anyway.

How are you dealing with these situations, are there better ways than this?

Cheers,
galo


Chris Hostetter wrote:
> : problems for a few weeks, snappuller and snapinstaller run every hour
> : normally, install the new index without errors etc.
> :
> : In the last couple of days it seems like I need to restart tomcat for
> : the new documents to appear in the slave indexes. New docs are updated,
> : commited and appear if I do a search on the master, but if after the
> : synchronization I search in the slaves they don't appear. If I restart
> : tomcat they do.
> 
> snappuller "notifies" the slaves that they should reopen the index by
> executing the bin/commit script ... if that fails it should log a message
> ... but maybe bin/commit is logging an error in it's log and exiting with
> a successful return code?
> 
> do you see the "commit" log messages from Solr?  do you see the /update
> requests in the slaves tomcat access logs?
> 
> 
> 
> 
> 
> -Hoss
> 
> 


-- 
Galo Navarro, Developer

galo@last.fm
t. +44 (0)20 7780 7080

Last.fm | http://www.last.fm
Karen House 1-11 Baches Street
London N1 6DL

http://www.last.fm/user/galeote

Re: New docs need server restart after synchronization

[Chris Hostetter <ho...@fucit.org>]

: problems for a few weeks, snappuller and snapinstaller run every hour
: normally, install the new index without errors etc.
:
: In the last couple of days it seems like I need to restart tomcat for
: the new documents to appear in the slave indexes. New docs are updated,
: commited and appear if I do a search on the master, but if after the
: synchronization I search in the slaves they don't appear. If I restart
: tomcat they do.

snappuller "notifies" the slaves that they should reopen the index by
executing the bin/commit script ... if that fails it should log a message
... but maybe bin/commit is logging an error in it's log and exiting with
a successful return code?

do you see the "commit" log messages from Solr?  do you see the /update
requests in the slaves tomcat access logs?





-Hoss