You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Barry Forrest <bf...@gmail.com> on 2007/11/12 00:16:28 UTC
Optimizing index takes too long
Hi,
Optimizing my index of 1.5 million documents takes days and days.
I have a collection of 10 million documents that I am trying to index
with Lucene. I've divided the collection into chunks of about 1.5 - 2
million documents each. Indexing 1.5 documents is fast enough (about
12 hours), but this results in an index directory containing about
35000 files. Optimizing this index takes several days, which is a bit
too long for my purposes. Each sub-index is about 150G.
What can I do to make this process faster?
Thanks for your help,
Barry
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Optimizing index takes too long
Posted by Barry Forrest <bf...@gmail.com>.
On Nov 12, 2007 1:15 PM, J.J. Larrea <jj...@panix.com> wrote:
>
> 2. Since the full document and its longer bibliographic subfields are
> being indexed but not stored, my guess is that the large size of the index
> segments is due to the inverted index rather than the stored data fields.
> But you can roughly verify by checking the size of the files in the index,
> with Luke's Files tab or simply an ls -l. For example .fdt files are stored
> data while .tis are the inverted index; see
> http://lucene.apache.org/java/docs/fileformats.html And if you have .cfs
> files...
OK here are some of the details of one index directory. The number of
indexed documents is approximately 1.5million.
$ ls -1 |wc -l
29542
$ du -hs .
90G .
$ // Show space used (in GB) by file extension.
$ for filetype in `ls -1 | sed -r "s/.*\.(.*)/\1/" | sort -u` ;\
do echo -n "filetype=$filetype: " ; (echo -n "scale=2; \
(" ; (for size in `ls -l *.$filetype| cut -c 24-34`; do \
echo -n "$size+"; done) ; echo "0)/10^9") | bc; done
filetype=fdt: .44
filetype=fdx: .01
filetype=fnm: .01
filetype=frq: 15.95
filetype=lock: 0
filetype=nrm: .03
filetype=prx: 70.94
filetype=tii: .12
filetype=tis: 8.79
So most of the space is occupied with .prx files.
Thanks
Barry
Re: Optimizing index takes too long
Posted by "J.J. Larrea" <jj...@panix.com>.
Hi. Here are a couple of thoughts:
1. Your problem description would be a little easier to parse if you didn't use the word "stored" to refer to fields which are not, in a Lucene sense, stored, only indexed. For example, one doesn't "store" stemmed and unstemmed versions, since stemming has absolutely no effect on the stored Documents (and here I am using the capitalized word to distinguish Lucene Documents from your source documents).
2. Since the full document and its longer bibliographic subfields are being indexed but not stored, my guess is that the large size of the index segments is due to the inverted index rather than the stored data fields. But you can roughly verify by checking the size of the files in the index, with Luke's Files tab or simply an ls -l. For example .fdt files are stored data while .tis are the inverted index; see http://lucene.apache.org/java/docs/fileformats.html And if you have .cfs files...
3. You have set MaxFieldLength to Integer.MAX_VALUE. Is there a specific requirement for that being unbounded? If you reduce the size, e.g. to 50k, you will dramatically reduce the size of the inverted index.
For fields for which norms will never be used (ie. queries of those fields affect hits but do not contribute to the score), disable them.
4. Make sure you have set useCompoundFile(false)! If it is true (which is the default), every round of optimization* writes separate per-role files, then as a separate step packs them up into a compound file. Besides causing an additional recopy, it means that optimization can take three times rather than twice the space on disk**.
5. 35000 files for 1.5M documents - that's <50 documents per file, way too low! When I index 27M documents I think it's a lot when I'm up to 100 files! Reduce MergeFactor, increase MaxMergeDocs. I think if you reduce MergeFactor from 50 to 10 and increase MaxMergeDocs from 2000 to 10000, you will end up with a similar memory footprint but a significantly more efficient disk footprint and far fewer rounds of optimization. Also, what about MinMergeDocs?
I've not experimented with the RAMBufferSizeMB parameter, but 32Mb seems low for an app dealing with such heavyweight documents. Perhaps someone else knows better.
Note that if useCompoundFiles is false, you will end up with ~8 times the number of files (depending on features such as term vectors, etc.), so it is essential to first reduce the number with MergeFactor and MaxMergeDocs.
Through some judicious combination of the above steps, I am confident you can greatly reduce indexing time, optimization time, and index size, without impairing the ability to meet functional requirements.
- J.J.
*I'm not absolutely sure it's still every round of optimization, but it's certainly the case for the final round.
**At least in Lucene 1.9, I'm not sure about 2.3
At 11:05 AM +1100 11/12/07, Barry Forrest wrote:
>Hi,
>
>Thanks for your help.
>
>I'm using Lucene 2.3.
>
>Raw document size is about 138G for 1.5M documents, which is about
>250k per document.
>
>IndexWriter settings are MergeFactor 50, MaxMergeDocs 2000,
>RAMBufferSizeMB 32, MaxFieldLength Integer.MAX_VALUE.
>
>Each document has about 10 short bibliographic fields and 3 longer
>content fields and 1 field that contains the entire contents of the
>document. The longer content fields are stored twice - in a stemmed
>and unstemmed form. So actually there are about 8 longer content
>fields. (The effect of storing stemmed and unstemmed versions is to
>approximately double the index size over storing the content only
>once). About half the short bibliographic fields are stored
>(compressed) in the index. The longer content fields are not stored,
>and no term vectors are stored.
>
>The hardware is quite new and fast: 8 cores, 15,000 RPM disks.
>
>Thanks again
>Barry
>
>On Nov 12, 2007 10:41 AM, Grant Ingersoll <gs...@apache.org> wrote:
>> Hmmm, something doesn't sound quite right. You have 10 million docs,
> > split into 5 or so indexes, right? And each sub index is 150
>> gigabytes? How big are your documents?
>>
>> Can you provide more info about what your Directory and IndexWriter
>> settings are? What version of Lucene are you using? What are your
>> Field settings? Are you storing info? What about Term Vectors?
>>
>> Can you explain more about your documents, etc? 10 million doesn't
>> sound like it would need to be split up that much, if at all,
>> depending on your hardware.
>>
>> The wiki has some excellent resources on improving both indexing and
>> search speed.
>>
>> -Grant
>>
>>
>>
>> On Nov 11, 2007, at 6:16 PM, Barry Forrest wrote:
>>
>> > Hi,
>> >
> > > Optimizing my index of 1.5 million documents takes days and days.
>> >
>> > I have a collection of 10 million documents that I am trying to index
>> > with Lucene. I've divided the collection into chunks of about 1.5 - 2
>> > million documents each. Indexing 1.5 documents is fast enough (about
>> > 12 hours), but this results in an index directory containing about
>> > 35000 files. Optimizing this index takes several days, which is a bit
>> > too long for my purposes. Each sub-index is about 150G.
>> >
>> > What can I do to make this process faster?
> > >
>> > Thanks for your help,
> > > Barry
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Optimizing index takes too long
Posted by Michael McCandless <lu...@mikemccandless.com>.
> I am using the 2.3-dev version only because LUCENE-843 suggested
> that this might be a path to faster indexing. I started out using
> 2.2 and can easily go back. I am using default MergePolicy and
> MergeScheduler.
Did you note any indexing or optimize speed differences between 2.2 &
2.3-dev?
One important thing to realize is LUCENE-843 only addresses speeding
up the creation of newly flushed segments (from add/updateDocument()
calls).
It does not speed up segment merging (which is what optimize() is
actually doing), though there have been at least a couple recent
issues on 2.3-dev that should speed up merging:
- LUCENE-1043 (use bulk byte-copying to merge stored fields when
possible)
- LUCENE-888 (increase buffer sizes in input/outputs used during
merging)
There is a separate issue open (LUCENE-856) to track ideas on how to
speed up segment merging.
> Also, maybe Mike M. can chime in w/ how compressed fields are merged
> now.
As far as I know, merging of compressed fields is unchanged wrt 2.2:
we still [efficiently] load & rewrite the raw bytes without
decompresssing them.
> For a start, I would lower the merge factor quite a bit. A high
> merge factor is over rated :)
I would second this one: try lower values and see if optimizing is
faster. It's not clear that a high mergeFactor gives faster merging
overall.
> The hardware is quite new and fast: 8 cores, 15,000 RPM disks.
Your machine sounds fabulous (I'm jealous!) so the numbers don't seem
to add up.
Are you giving the JVM plenty of RAM? And the machine is not
swapping? Indexing/optimizing should not be RAM intensive, like
searching, but it's still worth checking into.
> IndexWriter settings are MergeFactor 50, MaxMergeDocs 2000,
> RAMBufferSizeMB 32, MaxFieldLength Integer.MAX_VALUE.
The MaxMergeDocs=2000 is what's causing 35K files in your index (which
is far too many) -- and, this also foists all the merge cost into your
optimize call. With the default MaxMergeDocs (effectively unlimited)
Lucene would do more merging, concurrently (in 2.3-dev), as the
index is being built.
If possible, the next time you run optimize() could you also call
IndexWriter.setInfoStream(...) and post the result log? This would
have details on what merges are being selected, in case something
is going awry in the LogByteSizeMergePolicy.
Can you do "ls -l" on one of your sub-indices and post the results?
This would give us a raw check on where the bytes are going in the
index...
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Optimizing index takes too long
Posted by Barry Forrest <bf...@gmail.com>.
Thanks very much for all your suggestions.
I will work through these to see what works. Appreciate that indexing takes
many hours, so it will take me a few days. Working with a subset isn't
really indicative, since the problems only manifest with larger indexes.
(Note that this might be a solution to my problem - simply subdivide the
collection even further). I will reduce the merge factor and increase the
max merge docs.
No, I am not using the compound index format.
Apologies for my confusing use of the word "stored". Most fields are not
stored at all, although some are indexed twice.
I am using the 2.3-dev version only because LUCENE-843 suggested that this
might be a path to faster indexing. I started out using 2.2 and can easily
go back. I am using default MergePolicy and MergeScheduler.
The field constructions are something like the following.
// Stored bibliographic fields.
document.add(new Field(FieldName.FIELD1.name(), documentBean.getField1()
Field.Store.COMPRESS,
Field.Index.UN_TOKENIZED, Field.TermVector.NO));
...
// Unstored content fields - stemmed and unstemmed versions.
document.add(new Field(Fields.FIELD5_STEMMED.name(),
documentBean.getField5Stemmed() Field.Store.NO,
Field.Index.TOKENIZED, Field.TermVector.NO));
document.add(new Field(Fields.FIELD5_UNSTEMMED.name(),
documentBean.getField5Unstemmed() Field.Store.NO,Field.Index.TOKENIZED,
Field.TermVector.NO));
...
Let me do some more experiments and I'll report back.
Thanks
Barry.
On Nov 12, 2007 1:30 PM, Grant Ingersoll <gs...@apache.org> wrote:
> Not sure the numbers are off w/ documents that big, although I imagine
> you are hitting the token limit w/ docs that big. Is this all on one
> machine as you described, or are you saying you have a couple of
> these? If one, have you tried having just one index?
>
> Since you are using 2.3 (note to other readers, 2.3 is NOT released
> yet, he really means 2.3-dev), are which MergeScheduler and
> MergePolicy are you using? Can you do some profiling to see where is
> spending time?
>
> Also, maybe Mike M. can chime in w/ how compressed fields are merged
> now. I want to say that with the new indexing changes, they are all
> done right away and not revisited so that shouldn't be an issue.
> Having said that, I am a bit confused by some of your terminology.
> You say some Fields are stored twice, but then say they are not
> stored. Can you share what the actual Field constructions are?
> There probably isn't a reason to compress the short biblio fields.
> Lucene Field compression, while not deprecated, really isn't
> recommended, b/c it doesn't give the application much control (since
> it uses the highest level of compression and is not tunable.) The
> better approach is to do the compression yourself and store as a
> binary. Again, though, it doesn't sound like you need compression for
> those fields.
>
> Are you using compound file format or not?
>
> Also, were you using 2.2 before and upgraded, or is this an
> application built on 2.3 to begin with? If on 2.2, did you see these
> problems before?
>
> Cheers,
> Grant
>
>
> On Nov 11, 2007, at 8:49 PM, Mark Miller wrote:
>
> > For a start, I would lower the merge factor quite a bit. A high
> > merge factor is over rated :) You will build the index faster, but
> > searches will be slower and an optimize takes much longer.
> > Essentially, the time you save when indexing is paid when optimizing
> > anyway. You might as well amortize the cost with a lower merge factor.
> >
> > Grant seems to think the numbers are off anyway, so you may have
> > more to do -- just a suggestion about the merge factor. How much RAM
> > are you giving your application?
> >
> > With a machine with 8 cores and 15,000rpm, days does seem a little
> > ridiculous.
> >
> > - Mark
> >
> > Barry Forrest wrote:
> >> Hi,
> >>
> >> Thanks for your help.
> >>
> >> I'm using Lucene 2.3.
> >>
> >> Raw document size is about 138G for 1.5M documents, which is about
> >> 250k per document.
> >>
> >> IndexWriter settings are MergeFactor 50, MaxMergeDocs 2000,
> >> RAMBufferSizeMB 32, MaxFieldLength Integer.MAX_VALUE.
> >>
> >> Each document has about 10 short bibliographic fields and 3 longer
> >> content fields and 1 field that contains the entire contents of the
> >> document. The longer content fields are stored twice - in a stemmed
> >> and unstemmed form. So actually there are about 8 longer content
> >> fields. (The effect of storing stemmed and unstemmed versions is to
> >> approximately double the index size over storing the content only
> >> once). About half the short bibliographic fields are stored
> >> (compressed) in the index. The longer content fields are not stored,
> >> and no term vectors are stored.
> >>
> >> The hardware is quite new and fast: 8 cores, 15,000 RPM disks.
> >>
> >> Thanks again
> >> Barry
> >>
> >> On Nov 12, 2007 10:41 AM, Grant Ingersoll <gs...@apache.org>
> >> wrote:
> >>
> >>> Hmmm, something doesn't sound quite right. You have 10 million
> >>> docs,
> >>> split into 5 or so indexes, right? And each sub index is 150
> >>> gigabytes? How big are your documents?
> >>>
> >>> Can you provide more info about what your Directory and IndexWriter
> >>> settings are? What version of Lucene are you using? What are your
> >>> Field settings? Are you storing info? What about Term Vectors?
> >>>
> >>> Can you explain more about your documents, etc? 10 million doesn't
> >>> sound like it would need to be split up that much, if at all,
> >>> depending on your hardware.
> >>>
> >>> The wiki has some excellent resources on improving both indexing and
> >>> search speed.
> >>>
> >>> -Grant
> >>>
> >>>
> >>>
> >>> On Nov 11, 2007, at 6:16 PM, Barry Forrest wrote:
> >>>
> >>>
> >>>> Hi,
> >>>>
> >>>> Optimizing my index of 1.5 million documents takes days and days.
> >>>>
> >>>> I have a collection of 10 million documents that I am trying to
> >>>> index
> >>>> with Lucene. I've divided the collection into chunks of about
> >>>> 1.5 - 2
> >>>> million documents each. Indexing 1.5 documents is fast enough
> >>>> (about
> >>>> 12 hours), but this results in an index directory containing about
> >>>> 35000 files. Optimizing this index takes several days, which is
> >>>> a bit
> >>>> too long for my purposes. Each sub-index is about 150G.
> >>>>
> >>>> What can I do to make this process faster?
> >>>>
> >>>> Thanks for your help,
> >>>> Barry
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>>
> >>>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Optimizing index takes too long
Posted by Grant Ingersoll <gs...@apache.org>.
Not sure the numbers are off w/ documents that big, although I imagine
you are hitting the token limit w/ docs that big. Is this all on one
machine as you described, or are you saying you have a couple of
these? If one, have you tried having just one index?
Since you are using 2.3 (note to other readers, 2.3 is NOT released
yet, he really means 2.3-dev), are which MergeScheduler and
MergePolicy are you using? Can you do some profiling to see where is
spending time?
Also, maybe Mike M. can chime in w/ how compressed fields are merged
now. I want to say that with the new indexing changes, they are all
done right away and not revisited so that shouldn't be an issue.
Having said that, I am a bit confused by some of your terminology.
You say some Fields are stored twice, but then say they are not
stored. Can you share what the actual Field constructions are?
There probably isn't a reason to compress the short biblio fields.
Lucene Field compression, while not deprecated, really isn't
recommended, b/c it doesn't give the application much control (since
it uses the highest level of compression and is not tunable.) The
better approach is to do the compression yourself and store as a
binary. Again, though, it doesn't sound like you need compression for
those fields.
Are you using compound file format or not?
Also, were you using 2.2 before and upgraded, or is this an
application built on 2.3 to begin with? If on 2.2, did you see these
problems before?
Cheers,
Grant
On Nov 11, 2007, at 8:49 PM, Mark Miller wrote:
> For a start, I would lower the merge factor quite a bit. A high
> merge factor is over rated :) You will build the index faster, but
> searches will be slower and an optimize takes much longer.
> Essentially, the time you save when indexing is paid when optimizing
> anyway. You might as well amortize the cost with a lower merge factor.
>
> Grant seems to think the numbers are off anyway, so you may have
> more to do -- just a suggestion about the merge factor. How much RAM
> are you giving your application?
>
> With a machine with 8 cores and 15,000rpm, days does seem a little
> ridiculous.
>
> - Mark
>
> Barry Forrest wrote:
>> Hi,
>>
>> Thanks for your help.
>>
>> I'm using Lucene 2.3.
>>
>> Raw document size is about 138G for 1.5M documents, which is about
>> 250k per document.
>>
>> IndexWriter settings are MergeFactor 50, MaxMergeDocs 2000,
>> RAMBufferSizeMB 32, MaxFieldLength Integer.MAX_VALUE.
>>
>> Each document has about 10 short bibliographic fields and 3 longer
>> content fields and 1 field that contains the entire contents of the
>> document. The longer content fields are stored twice - in a stemmed
>> and unstemmed form. So actually there are about 8 longer content
>> fields. (The effect of storing stemmed and unstemmed versions is to
>> approximately double the index size over storing the content only
>> once). About half the short bibliographic fields are stored
>> (compressed) in the index. The longer content fields are not stored,
>> and no term vectors are stored.
>>
>> The hardware is quite new and fast: 8 cores, 15,000 RPM disks.
>>
>> Thanks again
>> Barry
>>
>> On Nov 12, 2007 10:41 AM, Grant Ingersoll <gs...@apache.org>
>> wrote:
>>
>>> Hmmm, something doesn't sound quite right. You have 10 million
>>> docs,
>>> split into 5 or so indexes, right? And each sub index is 150
>>> gigabytes? How big are your documents?
>>>
>>> Can you provide more info about what your Directory and IndexWriter
>>> settings are? What version of Lucene are you using? What are your
>>> Field settings? Are you storing info? What about Term Vectors?
>>>
>>> Can you explain more about your documents, etc? 10 million doesn't
>>> sound like it would need to be split up that much, if at all,
>>> depending on your hardware.
>>>
>>> The wiki has some excellent resources on improving both indexing and
>>> search speed.
>>>
>>> -Grant
>>>
>>>
>>>
>>> On Nov 11, 2007, at 6:16 PM, Barry Forrest wrote:
>>>
>>>
>>>> Hi,
>>>>
>>>> Optimizing my index of 1.5 million documents takes days and days.
>>>>
>>>> I have a collection of 10 million documents that I am trying to
>>>> index
>>>> with Lucene. I've divided the collection into chunks of about
>>>> 1.5 - 2
>>>> million documents each. Indexing 1.5 documents is fast enough
>>>> (about
>>>> 12 hours), but this results in an index directory containing about
>>>> 35000 files. Optimizing this index takes several days, which is
>>>> a bit
>>>> too long for my purposes. Each sub-index is about 150G.
>>>>
>>>> What can I do to make this process faster?
>>>>
>>>> Thanks for your help,
>>>> Barry
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Optimizing index takes too long
Posted by Mark Miller <ma...@gmail.com>.
For a start, I would lower the merge factor quite a bit. A high merge
factor is over rated :) You will build the index faster, but searches
will be slower and an optimize takes much longer. Essentially, the time
you save when indexing is paid when optimizing anyway. You might as well
amortize the cost with a lower merge factor.
Grant seems to think the numbers are off anyway, so you may have more to
do -- just a suggestion about the merge factor. How much RAM are you
giving your application?
With a machine with 8 cores and 15,000rpm, days does seem a little
ridiculous.
- Mark
Barry Forrest wrote:
> Hi,
>
> Thanks for your help.
>
> I'm using Lucene 2.3.
>
> Raw document size is about 138G for 1.5M documents, which is about
> 250k per document.
>
> IndexWriter settings are MergeFactor 50, MaxMergeDocs 2000,
> RAMBufferSizeMB 32, MaxFieldLength Integer.MAX_VALUE.
>
> Each document has about 10 short bibliographic fields and 3 longer
> content fields and 1 field that contains the entire contents of the
> document. The longer content fields are stored twice - in a stemmed
> and unstemmed form. So actually there are about 8 longer content
> fields. (The effect of storing stemmed and unstemmed versions is to
> approximately double the index size over storing the content only
> once). About half the short bibliographic fields are stored
> (compressed) in the index. The longer content fields are not stored,
> and no term vectors are stored.
>
> The hardware is quite new and fast: 8 cores, 15,000 RPM disks.
>
> Thanks again
> Barry
>
> On Nov 12, 2007 10:41 AM, Grant Ingersoll <gs...@apache.org> wrote:
>
>> Hmmm, something doesn't sound quite right. You have 10 million docs,
>> split into 5 or so indexes, right? And each sub index is 150
>> gigabytes? How big are your documents?
>>
>> Can you provide more info about what your Directory and IndexWriter
>> settings are? What version of Lucene are you using? What are your
>> Field settings? Are you storing info? What about Term Vectors?
>>
>> Can you explain more about your documents, etc? 10 million doesn't
>> sound like it would need to be split up that much, if at all,
>> depending on your hardware.
>>
>> The wiki has some excellent resources on improving both indexing and
>> search speed.
>>
>> -Grant
>>
>>
>>
>> On Nov 11, 2007, at 6:16 PM, Barry Forrest wrote:
>>
>>
>>> Hi,
>>>
>>> Optimizing my index of 1.5 million documents takes days and days.
>>>
>>> I have a collection of 10 million documents that I am trying to index
>>> with Lucene. I've divided the collection into chunks of about 1.5 - 2
>>> million documents each. Indexing 1.5 documents is fast enough (about
>>> 12 hours), but this results in an index directory containing about
>>> 35000 files. Optimizing this index takes several days, which is a bit
>>> too long for my purposes. Each sub-index is about 150G.
>>>
>>> What can I do to make this process faster?
>>>
>>> Thanks for your help,
>>> Barry
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Optimizing index takes too long
Posted by Barry Forrest <bf...@gmail.com>.
Hi,
Thanks for your help.
I'm using Lucene 2.3.
Raw document size is about 138G for 1.5M documents, which is about
250k per document.
IndexWriter settings are MergeFactor 50, MaxMergeDocs 2000,
RAMBufferSizeMB 32, MaxFieldLength Integer.MAX_VALUE.
Each document has about 10 short bibliographic fields and 3 longer
content fields and 1 field that contains the entire contents of the
document. The longer content fields are stored twice - in a stemmed
and unstemmed form. So actually there are about 8 longer content
fields. (The effect of storing stemmed and unstemmed versions is to
approximately double the index size over storing the content only
once). About half the short bibliographic fields are stored
(compressed) in the index. The longer content fields are not stored,
and no term vectors are stored.
The hardware is quite new and fast: 8 cores, 15,000 RPM disks.
Thanks again
Barry
On Nov 12, 2007 10:41 AM, Grant Ingersoll <gs...@apache.org> wrote:
> Hmmm, something doesn't sound quite right. You have 10 million docs,
> split into 5 or so indexes, right? And each sub index is 150
> gigabytes? How big are your documents?
>
> Can you provide more info about what your Directory and IndexWriter
> settings are? What version of Lucene are you using? What are your
> Field settings? Are you storing info? What about Term Vectors?
>
> Can you explain more about your documents, etc? 10 million doesn't
> sound like it would need to be split up that much, if at all,
> depending on your hardware.
>
> The wiki has some excellent resources on improving both indexing and
> search speed.
>
> -Grant
>
>
>
> On Nov 11, 2007, at 6:16 PM, Barry Forrest wrote:
>
> > Hi,
> >
> > Optimizing my index of 1.5 million documents takes days and days.
> >
> > I have a collection of 10 million documents that I am trying to index
> > with Lucene. I've divided the collection into chunks of about 1.5 - 2
> > million documents each. Indexing 1.5 documents is fast enough (about
> > 12 hours), but this results in an index directory containing about
> > 35000 files. Optimizing this index takes several days, which is a bit
> > too long for my purposes. Each sub-index is about 150G.
> >
> > What can I do to make this process faster?
> >
> > Thanks for your help,
> > Barry
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Optimizing index takes too long
Posted by Grant Ingersoll <gs...@apache.org>.
Hmmm, something doesn't sound quite right. You have 10 million docs,
split into 5 or so indexes, right? And each sub index is 150
gigabytes? How big are your documents?
Can you provide more info about what your Directory and IndexWriter
settings are? What version of Lucene are you using? What are your
Field settings? Are you storing info? What about Term Vectors?
Can you explain more about your documents, etc? 10 million doesn't
sound like it would need to be split up that much, if at all,
depending on your hardware.
The wiki has some excellent resources on improving both indexing and
search speed.
-Grant
On Nov 11, 2007, at 6:16 PM, Barry Forrest wrote:
> Hi,
>
> Optimizing my index of 1.5 million documents takes days and days.
>
> I have a collection of 10 million documents that I am trying to index
> with Lucene. I've divided the collection into chunks of about 1.5 - 2
> million documents each. Indexing 1.5 documents is fast enough (about
> 12 hours), but this results in an index directory containing about
> 35000 files. Optimizing this index takes several days, which is a bit
> too long for my purposes. Each sub-index is about 150G.
>
> What can I do to make this process faster?
>
> Thanks for your help,
> Barry
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
indexing a large number of fields (and nesting them)
Posted by Se...@emc.com.
Hi,
I need to index tons of meta-data fields along with every document
(around 80 fields, mostly strings of 32 characters, some integers, bools
and dates too, a couple of strings are longer like 64 chars or 120).
Also it would be nice if there was a way to represent nested fields, and
query for them. Right now nesting is handled by redundantly creating
fields. And numbers and dates are normalized to strings during
indexing/querying.
So representing the following meta data:
<field A>
<sub field a>
<sub field b>
</field A>
in lucene looks like:
Document.add(new Field("fieldA", "value of entire field A section",
Field.Store.YES, Field.Index.TOKENIZED)
Document.add(new Field("fielda", "value of field a", Field.Store.YES,
Field.Index.TOKENIZED)
Document.add(new Field("fielda", "value of field b", Field.Store.YES,
Field.Index.TOKENIZED)
Is there a better way to do this? I'd appreciate any suggestions.
Thanks,
Yasoja
PS: Current implementation is on an older lucene 2.1 but moving to 2.2
or 2.3 is not a problem.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Optimizing index takes too long
Posted by Eric Louvard <er...@hauk-sasko.de>.
You could have a look at this thread.
http://www.gossamer-threads.com/lists/lucene/java-user/29354
regards.
Barry Forrest schrieb:
> Hi,
>
> Optimizing my index of 1.5 million documents takes days and days.
>
> I have a collection of 10 million documents that I am trying to index
> with Lucene. I've divided the collection into chunks of about 1.5 - 2
> million documents each. Indexing 1.5 documents is fast enough (about
> 12 hours), but this results in an index directory containing about
> 35000 files. Optimizing this index takes several days, which is a bit
> too long for my purposes. Each sub-index is about 150G.
>
> What can I do to make this process faster?
>
> Thanks for your help,
> Barry
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
--
Mit freundlichen Grüßen
i. A. Éric Louvard
HAUK & SASKO Ingenieurgesellschaft mbH
Zettachring 2
D-70567 Stuttgart
Phone: +49 7 11 7 25 89 - 19
Fax: +49 7 11 7 25 89 - 50
E-Mail: eric.louvard@hauk-sasko.de
www: www.hauk-sasko.de
Niederlassung Stuttgart
Firmensitz: Markstr. 77, 44801 Bochum
Registergericht: Amtsgericht Bochum, HRB 2532
Geschäftsführer: Dr.-Ing. Pavol Sasko
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Optimizing index takes too long
Posted by Lucene User <lu...@gmail.com>.
what type of documents are indexing
regards
gaurav
On 11/11/07, Barry Forrest <bf...@gmail.com> wrote:
>
> Hi,
>
> Optimizing my index of 1.5 million documents takes days and days.
>
> I have a collection of 10 million documents that I am trying to index
> with Lucene. I've divided the collection into chunks of about 1.5 - 2
> million documents each. Indexing 1.5 documents is fast enough (about
> 12 hours), but this results in an index directory containing about
> 35000 files. Optimizing this index takes several days, which is a bit
> too long for my purposes. Each sub-index is about 150G.
>
> What can I do to make this process faster?
>
> Thanks for your help,
> Barry
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>