You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Tim Mahy <ti...@infosupport.com> on 2008/05/06 13:56:48 UTC

RE: Delete's increase while adding new documents

Hi all,

it seems that we get errors during the auto-commit :


java.io.FileNotFoundException: /opt/solr/upload/nl/archive/data/index/_4x.fnm (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.&lt;init&gt;(RandomAccessFile.java:212)
        at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.&lt;init&gt;(FSDirectory.java:501)
        at org.apache.lucene.store.FSDirectory$FSIndexInput.&lt;init&gt;(FSDirectory.java:526)

the _4x.fnm file is not on the file system. When we switch from autocommit to manual commits throughout xml messages we get the same kind of errors.
Any idea what could be wrong in our configuration to cause these exceptions ?

Greetings,
Tim
________________________________________
Van: Tim Mahy [timm@infosupport.com]
Verzonden: maandag 28 april 2008 12:11
Aan: solr-user@lucene.apache.org
Onderwerp: RE: Delete's increase while adding new documents

Hi all,

thank you for your reply. The id's that we send are unique, so we still have no clue what is happening :)

greetings,
Tim

-----Oorspronkelijk bericht-----
Van: Mike Klaas [mailto:mike.klaas@gmail.com]
Verzonden: za 26-4-2008 1:52
Aan: solr-user@lucene.apache.org
Onderwerp: Re: Delete's increase while adding new documents

On 25-Apr-08, at 4:27 AM, Tim Mahy wrote:
>
> Hi all,
>
> we send xml add document messages to Solr and we notice something
> very strange.
> We autocommit at 100000 documents, starting from a total clean index
> (removed the data folder), when we start uploading we notice that
> the docsPending is going up but also that the deletesPending is
> going up very fast. After reaching the first 100000 we queried to
> solr to return everything and the total results count was not 100000
> but somewhere around 77000 which is exactly 100000 - docsDeleted
> from the stats page.
>
> We used that Solr instance before, so my question is : is it
> possible that Solr remembers the unique identities somewhere else as
> in the data folder ? Btw we stopped Solr, removed the data folder
> and restarted Solr and than this behavior began...

Are you sure that all the documents you added were unique?

(btw, deletePending doesn't necessarily mean that an old version of
the doc was in the index, I think).

-Mike



Op dit e-mail bericht is de disclaimer van Info Support van toepassing, zie http://www.infosupport.nl/disclaimer

[cid:60561037AEC348669B4CF2083E6168F4]


Re: Delete's increase while adding new documents

Posted by Mike Klaas <mi...@gmail.com>.
Hi Tim,

You definitely have something seriously wrong with your setup--some  
people have reported thousands of documents indexed per second, and  
I've personal indexed millions of documents sans <commit/>.  We've  
never had a report of documents not being in the index despite being  
sent to Solr, nor missing index files which weren't caused by  
corruption (usually related to sudden power failure).

How many simultaneous threads are you using to post to Solr?  What  
version of Solr are you using?  Is the method you are using to post to  
Solr timing out and resending?

I'm afraid without a _lot_ more detail on how you're using Solr, it is  
difficult to us to help you.

-Mike

On 7-May-08, at 2:15 AM, Tim Mahy wrote:

> Hi all,
>
> it seems that we just post to much to fast to Solr.
>
> When we post 100 documents (seperate calls) and perform a commit  
> everything goes well, but as soon as we start sending thousands of  
> documents and than use autocommit or send the commit message we have  
> the situation that there are a lot of documents not in the index  
> although they were sended to Solr ...
>
> has anyone experience with how much documents you can import and at  
> which speed so that Solr stays stable ?
>
> We use Tomcat 5.5 and our java memory limit is 2gb.
>
> Greetings,
> Tim
> ________________________________________
> Van: Mike Klaas [mike.klaas@gmail.com]
> Verzonden: dinsdag 6 mei 2008 20:17
> Aan: solr-user@lucene.apache.org
> Onderwerp: Re: Delete's increase while adding new documents
>
> On 6-May-08, at 4:56 AM, Tim Mahy wrote:
>
>> Hi all,
>>
>> it seems that we get errors during the auto-commit :
>>
>>
>> java.io.FileNotFoundException: /opt/solr/upload/nl/archive/data/
>> index/_4x.fnm (No such file or directory)
>>       at java.io.RandomAccessFile.open(Native Method)
>>       at java.io.RandomAccessFile.&lt;init&gt;
>> (RandomAccessFile.java:212)
>>       at org.apache.lucene.store.FSDirectory$FSIndexInput
>> $Descriptor.&lt;init&gt;(FSDirectory.java:501)
>>       at org.apache.lucene.store.FSDirectory
>> $FSIndexInput.&lt;init&gt;(FSDirectory.java:526)
>>
>> the _4x.fnm file is not on the file system. When we switch from
>> autocommit to manual commits throughout xml messages we get the same
>> kind of errors.
>> Any idea what could be wrong in our configuration to cause these
>> exceptions ?
>
> I have only heard of that error appearing in two cases.  Either the
> index is corrupt, or something else deleted the file.  Are you sure
> that there is only one Solr instance that accesses the directory, and
> that nothing else ever touches it?
>
> Can you reproduce the deletion issue with a small number of documents
> (something that could be tested by one of us)?
>
> -Mike
>
>
>
>
> Please see our disclaimer, http://www.infosupport.be/Pages/Disclaimer.aspx


RE: Delete's increase while adding new documents

Posted by Tim Mahy <ti...@infosupport.com>.
Hi all,

it seems that we just post to much to fast to Solr.

When we post 100 documents (seperate calls) and perform a commit everything goes well, but as soon as we start sending thousands of documents and than use autocommit or send the commit message we have the situation that there are a lot of documents not in the index although they were sended to Solr ...

has anyone experience with how much documents you can import and at which speed so that Solr stays stable ?

We use Tomcat 5.5 and our java memory limit is 2gb.

Greetings,
Tim
________________________________________
Van: Mike Klaas [mike.klaas@gmail.com]
Verzonden: dinsdag 6 mei 2008 20:17
Aan: solr-user@lucene.apache.org
Onderwerp: Re: Delete's increase while adding new documents

On 6-May-08, at 4:56 AM, Tim Mahy wrote:

> Hi all,
>
> it seems that we get errors during the auto-commit :
>
>
> java.io.FileNotFoundException: /opt/solr/upload/nl/archive/data/
> index/_4x.fnm (No such file or directory)
>        at java.io.RandomAccessFile.open(Native Method)
>        at java.io.RandomAccessFile.&lt;init&gt;
> (RandomAccessFile.java:212)
>        at org.apache.lucene.store.FSDirectory$FSIndexInput
> $Descriptor.&lt;init&gt;(FSDirectory.java:501)
>        at org.apache.lucene.store.FSDirectory
> $FSIndexInput.&lt;init&gt;(FSDirectory.java:526)
>
> the _4x.fnm file is not on the file system. When we switch from
> autocommit to manual commits throughout xml messages we get the same
> kind of errors.
> Any idea what could be wrong in our configuration to cause these
> exceptions ?

I have only heard of that error appearing in two cases.  Either the
index is corrupt, or something else deleted the file.  Are you sure
that there is only one Solr instance that accesses the directory, and
that nothing else ever touches it?

Can you reproduce the deletion issue with a small number of documents
(something that could be tested by one of us)?

-Mike




Please see our disclaimer, http://www.infosupport.be/Pages/Disclaimer.aspx 

Re: Delete's increase while adding new documents

Posted by Mike Klaas <mi...@gmail.com>.
On 6-May-08, at 4:56 AM, Tim Mahy wrote:

> Hi all,
>
> it seems that we get errors during the auto-commit :
>
>
> java.io.FileNotFoundException: /opt/solr/upload/nl/archive/data/ 
> index/_4x.fnm (No such file or directory)
>        at java.io.RandomAccessFile.open(Native Method)
>        at java.io.RandomAccessFile.&lt;init&gt; 
> (RandomAccessFile.java:212)
>        at org.apache.lucene.store.FSDirectory$FSIndexInput 
> $Descriptor.&lt;init&gt;(FSDirectory.java:501)
>        at org.apache.lucene.store.FSDirectory 
> $FSIndexInput.&lt;init&gt;(FSDirectory.java:526)
>
> the _4x.fnm file is not on the file system. When we switch from  
> autocommit to manual commits throughout xml messages we get the same  
> kind of errors.
> Any idea what could be wrong in our configuration to cause these  
> exceptions ?

I have only heard of that error appearing in two cases.  Either the  
index is corrupt, or something else deleted the file.  Are you sure  
that there is only one Solr instance that accesses the directory, and  
that nothing else ever touches it?

Can you reproduce the deletion issue with a small number of documents  
(something that could be tested by one of us)?

-Mike