You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by antony jospeh <an...@gmail.com> on 2011/11/10 19:40:59 UTC
Improving indexing speed
Hi all,
I have a large number of files in a directory need to be index them. All
the files are in specific format need to parse to extract information after
that i had to index.
Single thread process one file at a time then i decided to use multi
threads when the main thread that loops the directory and pass the file
into pool of worker threads using a queue
all of the which share same index writer, How ever there is no any
significant changes in indexing speed
Any hints I am doing wrong or any suggestion
Thanks
Antony
Re: Improving indexing speed
Posted by Wouter Heijke <wh...@xs4all.nl>.
Hi,
We faced a similar problem.
The solution was to give the indexer less work and let worker threads do
all the work. They would result in pre-processed/analyzed/tokenized
Documents that could be indexed by the writer without any processing.
Wouter
> Hi
>
> the file to be indexed depends on the type of Document / data extractor
> ....
>
> My Document types are usually XML type and every time 2+ Million XML's
> are indexed and time taken is less then 5 minuts.
>
>
>
>
> with regards
> karthik
>
> On Fri, Nov 11, 2011 at 1:17 AM, Ian Lea <ia...@gmail.com> wrote:
>
>> And how long does it take just to read and parse the files, without
>> indexing them? Often that is the problem - nothing to do with lucene.
>>
>> There is plenty of good advice in
>> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed. A good match
>> on the subject of your message!
>>
>> --
>> Ian.
>>
>>
>> On Thu, Nov 10, 2011 at 7:22 PM, Simon Willnauer
>> <si...@googlemail.com> wrote:
>> > can you provide more information about your setup? things like how
>> > much time does it take to index you documents, how many docs do you
>> > index, what are your index writer settings, how many cores do you
>> > have, where do you read from and write to (disks). oh and what version
>> > of lucene are you using?
>> >
>> > thanks,
>> >
>> > simon
>> >
>> > On Thu, Nov 10, 2011 at 10:40 AM, antony jospeh
>> > <an...@gmail.com> wrote:
>> >> Hi all,
>> >>
>> >> I have a large number of files in a directory need to be index them.
>> All
>> >> the files are in specific format need to parse to extract information
>> after
>> >> that i had to index.
>> >> Single thread process one file at a time then i decided to use multi
>> >> threads when the main thread that loops the directory and pass the
>> file
>> >> into pool of worker threads using a queue
>> >> all of the which share same index writer, How ever there is no any
>> >> significant changes in indexing speed
>> >>
>> >> Any hints I am doing wrong or any suggestion
>> >>
>> >>
>> >> Thanks
>> >> Antony
>> >>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Improving indexing speed
Posted by KARTHIK SHIVAKUMAR <ns...@gmail.com>.
Hi
the file to be indexed depends on the type of Document / data extractor ....
My Document types are usually XML type and every time 2+ Million XML's
are indexed and time taken is less then 5 minuts.
with regards
karthik
On Fri, Nov 11, 2011 at 1:17 AM, Ian Lea <ia...@gmail.com> wrote:
> And how long does it take just to read and parse the files, without
> indexing them? Often that is the problem - nothing to do with lucene.
>
> There is plenty of good advice in
> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed. A good match
> on the subject of your message!
>
> --
> Ian.
>
>
> On Thu, Nov 10, 2011 at 7:22 PM, Simon Willnauer
> <si...@googlemail.com> wrote:
> > can you provide more information about your setup? things like how
> > much time does it take to index you documents, how many docs do you
> > index, what are your index writer settings, how many cores do you
> > have, where do you read from and write to (disks). oh and what version
> > of lucene are you using?
> >
> > thanks,
> >
> > simon
> >
> > On Thu, Nov 10, 2011 at 10:40 AM, antony jospeh
> > <an...@gmail.com> wrote:
> >> Hi all,
> >>
> >> I have a large number of files in a directory need to be index them. All
> >> the files are in specific format need to parse to extract information
> after
> >> that i had to index.
> >> Single thread process one file at a time then i decided to use multi
> >> threads when the main thread that loops the directory and pass the file
> >> into pool of worker threads using a queue
> >> all of the which share same index writer, How ever there is no any
> >> significant changes in indexing speed
> >>
> >> Any hints I am doing wrong or any suggestion
> >>
> >>
> >> Thanks
> >> Antony
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
*N.S.KARTHIK
R.M.S.COLONY
BEHIND BANK OF INDIA
R.M.V 2ND STAGE
BANGALORE
560094*
Re: Improving indexing speed
Posted by Ian Lea <ia...@gmail.com>.
And how long does it take just to read and parse the files, without
indexing them? Often that is the problem - nothing to do with lucene.
There is plenty of good advice in
http://wiki.apache.org/lucene-java/ImproveIndexingSpeed. A good match
on the subject of your message!
--
Ian.
On Thu, Nov 10, 2011 at 7:22 PM, Simon Willnauer
<si...@googlemail.com> wrote:
> can you provide more information about your setup? things like how
> much time does it take to index you documents, how many docs do you
> index, what are your index writer settings, how many cores do you
> have, where do you read from and write to (disks). oh and what version
> of lucene are you using?
>
> thanks,
>
> simon
>
> On Thu, Nov 10, 2011 at 10:40 AM, antony jospeh
> <an...@gmail.com> wrote:
>> Hi all,
>>
>> I have a large number of files in a directory need to be index them. All
>> the files are in specific format need to parse to extract information after
>> that i had to index.
>> Single thread process one file at a time then i decided to use multi
>> threads when the main thread that loops the directory and pass the file
>> into pool of worker threads using a queue
>> all of the which share same index writer, How ever there is no any
>> significant changes in indexing speed
>>
>> Any hints I am doing wrong or any suggestion
>>
>>
>> Thanks
>> Antony
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Improving indexing speed
Posted by Simon Willnauer <si...@googlemail.com>.
can you provide more information about your setup? things like how
much time does it take to index you documents, how many docs do you
index, what are your index writer settings, how many cores do you
have, where do you read from and write to (disks). oh and what version
of lucene are you using?
thanks,
simon
On Thu, Nov 10, 2011 at 10:40 AM, antony jospeh
<an...@gmail.com> wrote:
> Hi all,
>
> I have a large number of files in a directory need to be index them. All
> the files are in specific format need to parse to extract information after
> that i had to index.
> Single thread process one file at a time then i decided to use multi
> threads when the main thread that loops the directory and pass the file
> into pool of worker threads using a queue
> all of the which share same index writer, How ever there is no any
> significant changes in indexing speed
>
> Any hints I am doing wrong or any suggestion
>
>
> Thanks
> Antony
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org