changes in Lucene not visible through Solr

[Kaan Erdener <ke...@rackspace.com>]

hello,

I'm pulling data into Lucene several times an hour, approaching a  
total document count of  ~2 million. Sometimes I pull in brand new  
data, other times I replace an existing document with an updated  
copy. The number of documents that I update in Lucene will pretty  
much never be more than a thousand or so.

I have a Solr interface exposed to another part of our system, and  
it's basically sitting on top of Lucene as a read-only view into the  
index. I can perform updates and optimizations in Lucene and Solr  
will keep searching just fine, but I've discovered that changes in  
Lucene are not visible through Solr.

For example, say there is no matching document in Lucene for id=1000,  
so if I query Solr using id:1000, I will correctly find 0 matches.  
But then I import new data into Lucene, pulling in a new document  
where id=1000. At that point, the query for id:1000 should find one  
match, but it doesn't (0 still). If I bounce Solr, I can see the  
results just fine.

I thought it might be a caching issue, but I have all of the cache  
options disabled in solfconfig.xml and the problem persists. I also  
ran Lucene optimization while Solr was running, but again no fix. If  
anyone has any suggestions for configuring / poking Solr somehow so  
that it will see new changes in Lucene, please let me know.

cheers,
Kaan

Re: changes in Lucene not visible through Solr

[Chris Hostetter <ho...@fucit.org>]

: This worked great:
: 	curl http://localhost:8983/solr/update --data-binary '<commit/>'

you should go turn your caching options back on ... they'll help out a lot
if you tend to get any repetition in your queries, and they are "smart"
caches ... entries will expire on commit, and with autoWarming configured
Solr will even prefill the new caches on each commit based on the "best"
keys from the old cache.

: >> I thought it might be a caching issue, but I have all of the cache
: >> options disabled in solfconfig.xml and the problem persists. I also


-Hoss

Re: changes in Lucene not visible through Solr

[Kaan Erdener <ke...@rackspace.com>]

I'm glad I asked. I probably wouldn't have discovered that on my  
own... :)

This worked great:
	curl http://localhost:8983/solr/update --data-binary '<commit/>'

thanks,
Kaan

On Nov 29, 2006, at 12:31 AM, Mike Klaas wrote:

> On 11/28/06, Kaan Erdener <ke...@rackspace.com> wrote:
>
>> I thought it might be a caching issue, but I have all of the cache
>> options disabled in solfconfig.xml and the problem persists. I also
>> ran Lucene optimization while Solr was running, but again no fix. If
>> anyone has any suggestions for configuring / poking Solr somehow so
>> that it will see new changes in Lucene, please let me know.
>
> Changes to the lucene index are not visible until you perform
> '<commit/>'.  This is true regardless of whether you are modifying the
> index directly or through solr's xml interface.
>
> regards,
> -MIke

Re: changes in Lucene not visible through Solr

[Mike Klaas <mi...@gmail.com>]

On 11/28/06, Kaan Erdener <ke...@rackspace.com> wrote:

> I thought it might be a caching issue, but I have all of the cache
> options disabled in solfconfig.xml and the problem persists. I also
> ran Lucene optimization while Solr was running, but again no fix. If
> anyone has any suggestions for configuring / poking Solr somehow so
> that it will see new changes in Lucene, please let me know.

Changes to the lucene index are not visible until you perform
'<commit/>'.  This is true regardless of whether you are modifying the
index directly or through solr's xml interface.

regards,
-MIke