indexing performance

[Saravana <ms...@gmail.com>]

Hi,

Is it possible to scale lucene indexing like 2000/3000 documents per
second?  I need to index 10 fields each with 20 bytes long.  I should be
able to search by just giving any of the field values as criteria. I need to
get the count that has same field values.

Will it be possible?

with regards,
MSK

Re: indexing performance

[Chris Hostetter <ho...@fucit.org>]

:
: > I am trying to index the syslogs generated from one of my busy ftp
: > server so
: > that I can get counts specific to an user with the given time
: > frame. Since

: My immediate thought when reading this is if it really is a text
: search engine you want to use for this?

ditto ... if you aren't going to be using Lucene for anything more then
lookups by username finding yourself an extremely fast database that
doens't have the overhead of building a completley normalized index may be
a better way to go.

likewise if all you *really* need is counts per user withint buckets of
time, a 10 line app thta tails your log file and incriments in memory
counters flushing them to disk every few minutes would probably work to...


	...it's all a question of details.


-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: indexing performance

[karl wettin <ka...@gmail.com>]

27 feb 2007 kl. 16.49 skrev Saravana:

> I am trying to index the syslogs generated from one of my busy ftp  
> server so
> that I can get counts specific to an user with the given time  
> frame. Since
> my  ftp server is very busy it can generate so much syslogs per  
> second. And
> the important point here is I do not need any ranking etc..

My immediate thought when reading this is if it really is a text  
search engine you want to use for this?

If it is, how do you process the log? What fields to you split it up  
in and what analyzsers do you use? Optimizing the use of that is  
probably you best way to achieve better speed when appending  
documents to the index.

-- 
karl

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: indexing performance

[Saravana <ms...@gmail.com>]

Hi,

I thought of getting the maximum indexing rate by lucene. However I did the
test with sample strings and I am getting close to 600 documents/sec in a
512 MB RAM with 1.9 GHz Linux machine. Searching is pretty fast and I can
create new index files based on user or based on time etc so that I will end
up in searching with multiple index files.

I am trying to index the syslogs generated from one of my busy ftp server so
that I can get counts specific to an user with the given time frame. Since
my  ftp server is very busy it can generate so much syslogs per second. And
the important point here is I do not need any ranking etc..

So anybody faced the above scenario and any suggestions would be helpful.  I
am just trying to get the maximum performance from lucene for the given
hardware then i can purchase the needed hardware.

MSK


On 2/27/07, Erick Erickson <er...@gmail.com> wrote:
>
> How do you expect anyone to be able to answer such an open-ended
> question? What I'd do is create a test harness that generates a random
> set of strings and try it.
>
> Off the top of my head, this seems like a pretty steep requirement. And
> at 2,000 docs a second you're going to have a huge index pretty soon
> so you won't be able to do this for very long.
>
> What are you trying to accomplish anyway?
>
> Erick
>
> On 2/27/07, Saravana <ms...@gmail.com> wrote:
> >
> > Hi,
> >
> > Is it possible to scale lucene indexing like 2000/3000 documents per
> > second?  I need to index 10 fields each with 20 bytes long.  I should be
> > able to search by just giving any of the field values as criteria. I
> need
> > to
> > get the count that has same field values.
> >
> > Will it be possible?
> >
> > with regards,
> > MSK
> >
>

Re: indexing performance

[Erick Erickson <er...@gmail.com>]

How do you expect anyone to be able to answer such an open-ended
question? What I'd do is create a test harness that generates a random
set of strings and try it.

Off the top of my head, this seems like a pretty steep requirement. And
at 2,000 docs a second you're going to have a huge index pretty soon
so you won't be able to do this for very long.

What are you trying to accomplish anyway?

Erick

On 2/27/07, Saravana <ms...@gmail.com> wrote:
>
> Hi,
>
> Is it possible to scale lucene indexing like 2000/3000 documents per
> second?  I need to index 10 fields each with 20 bytes long.  I should be
> able to search by just giving any of the field values as criteria. I need
> to
> get the count that has same field values.
>
> Will it be possible?
>
> with regards,
> MSK
>

Re: indexing performance

[Nadav Har'El <ny...@math.technion.ac.il>]

On Tue, Feb 27, 2007, Saravana wrote about "indexing performance":
> Hi,
> 
> Is it possible to scale lucene indexing like 2000/3000 documents per
> second?

I don't know about the actual numbers, but one trick I've used in the past
to get really fast indexing was to create several independent indexes in
parallel. Simply, if you have, say, 4 CPUs and perhaps even several physical
disks, run 4 indexing processes each indexing a 1/4 of the files and creating
a separate index (on separate disks on separate IO channels, if possible).

At the end, you have 4 indexes which you can actually search together without
any real need to merge them, unless query performance is very important to
you as well.

> I need to index 10 fields each with 20 bytes long.  I should be
> able to search by just giving any of the field values as criteria. I need to
> get the count that has same field values.

You need just the counts? And you want to do just whole-field matching, not
word matching? In that case, Lucene might be an overkill for you. Or, if you
do use Lucene, make sure to use "keyword" (untokenized) fields, not
"tokenized" fields.

-- 
Nadav Har'El                        |      Thursday, Mar  1 2007, 11 Adar 5767
IBM Haifa Research Lab              |-----------------------------------------
                                    |Open your arms to change, but don't let
http://nadav.harel.org.il           |go of your values.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org