problem indexing large document collction on windows xp

[Thilo Will <t....@linguatec.de>]

Hello

I encoutered a problem when i tried to index large document collections
(about 20 mio documents).
The indexing failed with the IOException:

"Cannot delete deletables"

I tried different times (with the same document collection) and allways
received the error, but after a different number
of documents.

The exception is thrown after failing to delete the specfied file at
line 212 in FSDirectory.java.
I found the following cure:

after the lines
>   if (nu.exists())
>      if (!nu.delete()){

i replaced
>   throw new IOException("Cannot delete " + to);

with
>        while(nu.exists()){
>            nu.delete();
>            System.out.println("delete loop");
>            try {
>                Thread.sleep(5000);
>            } catch (InterruptedException e) {
>                throw new RuntimeException(e);
>            }

That is, now i retry deleting the file until it is successful.

After the changes, i was able to index all documents.

>From the fact that i observed several times

  "delete loop"
  
on the output console, it can be deduced that the 
body of the while loop was reached (and left) several times.


I am running lucene on windows xp.

Regards
Thilo

            


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: problem indexing large document collction on windows xp

[Bernhard Messer <be...@intrafind.de>]

Thilo,

thanks for your effort. Could you please open a new entry in Bugzilla, 
mark it as [PATCH] and add the diff file with your changes. This ensures 
that the sources and the information will not get lost in the huge 
universe of mailing lists. As soon there is time, one of the comitters 
will review and decide if it should be committed.


Bernhard

>Hello
>
>I encoutered a problem when i tried to index large document collections
>(about 20 mio documents).
>The indexing failed with the IOException:
>
>"Cannot delete deletables"
>
>I tried different times (with the same document collection) and allways
>received the error, but after a different number
>of documents.
>
>The exception is thrown after failing to delete the specfied file at
>line 212 in FSDirectory.java.
>I found the following cure:
>
>after the lines
>  
>
>>  if (nu.exists())
>>     if (!nu.delete()){
>>    
>>
>
>i replaced
>  
>
>>  throw new IOException("Cannot delete " + to);
>>    
>>
>
>with
>  
>
>>       while(nu.exists()){
>>           nu.delete();
>>           System.out.println("delete loop");
>>           try {
>>               Thread.sleep(5000);
>>           } catch (InterruptedException e) {
>>               throw new RuntimeException(e);
>>           }
>>    
>>
>
>That is, now i retry deleting the file until it is successful.
>
>After the changes, i was able to index all documents.
>
>>From the fact that i observed several times
>
>  "delete loop"
>  
>on the output console, it can be deduced that the 
>body of the while loop was reached (and left) several times.
>
>
>I am running lucene on windows xp.
>
>Regards
>Thilo
>
>            
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>  
>