Problems on massive indexing

[Omar Cal <om...@adriacom.it>]

Hello, i'm a newby of Lucene.

I've the following scenario:
-450.000 xml files and text files
-5 indexes, two stored and three unstored
-lucene library 1.2 (tested also 1.3RC)

When i try to index the material i've an IndexOutOfBoundException in the 
call to the index.optimize() after two hours of indexing.I know there is 
the bug 14355 and i think it could be the responsable for that exception.

I've tried also to index the whole material in subsequent runs but the 
problem seems to depend on the number of the documents.

I've tried to set the maxFieldLength at its maximum but nothing appened.

If i split the material in "trunks" of about 20.000 - 30.000 documents 
in each directory, the problem doesn't appear. Obviously i've to repeat 
the searches for each "trunk" (directory).

Anyone out there with a similar scenario? Other solutions?

Thanks, Omar



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Problems on massive indexing

[Kristian Hermsdorf <kr...@ifbus.de>]

Hi
 
I also got the IndexOutOfBoundException while optimizing the index (index- 
size about 1GB, 50 Docs with 25 fields each).
(optimizing was called via merging of RamDirectoy to FSDirectory).
The problem was that the FieldsReader tried to read more fields than 
existed ... .I've no glue how to fix it ...
 
bye
 
Kristian

On Tue, 01 Apr 2003 08:55:23 +0200, Omar Cal <om...@adriacom.it> wrote:

> Hello, i'm a newby of Lucene.
>
> I've the following scenario:
> -450.000 xml files and text files
> -5 indexes, two stored and three unstored
> -lucene library 1.2 (tested also 1.3RC)
>
> When i try to index the material i've an IndexOutOfBoundException in the 
> call to the index.optimize() after two hours of indexing.I know there is 
> the bug 14355 and i think it could be the responsable for that exception.
>
> I've tried also to index the whole material in subsequent runs but the 
> problem seems to depend on the number of the documents.
>
> I've tried to set the maxFieldLength at its maximum but nothing appened.
>
> If i split the material in "trunks" of about 20.000 - 30.000 documents in 
> each directory, the problem doesn't appear. Obviously i've to repeat the 
> searches for each "trunk" (directory).
>
> Anyone out there with a similar scenario? Other solutions?
>
> Thanks, Omar
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
>



-- 
ACRONYM: Acronym Causing Recursion, Obviously Numbing Your Mind  

Kristian Hermsdorf

interface:projects gmbh		
Tollkewitzer Straße  49		
01277 Dresden			


tel.: ++49-351-3 18 09 39

mail: Kristian.Hermsdorf@interface-business.de
priv: kristian@entropus.de

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org