You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Mark Miller <ma...@gmail.com> on 2006/08/12 00:10:31 UTC
30 milllion+ docs on a single server
I've made a nice little archive application with lucene. I made it to
handle our largest need: 2.5 million docs or so on a single server. Now
the powers that be say: lets use it for a 30+ million document archive
on a single server! (each doc size maybe 10k max...as small as a 1 or
2k) Please tell me why we are in trouble...please tell me why we are
not. I have tested up to 2 million docs without much trouble but 30
million...the average search will include a sort on a field as
well...can I search 30+ million docs with a sort? Man am I worried about
that. Maybe the server will have 8 procs and 12 billion gigs of RAM.
Mabye. Even still, Tomcat seems to be able to launch with a max of 1.5
or 1.6 gig of Ram in Windows. What do you think? 30 million+ sounds like
too much of a load to me for a single server. Not that they care what I
think...I only wrote the thing (man I hate my job, offer me a new one :)
)...please...comments?
Cheers,
Miserable Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 30 milllion+ docs on a single server
Posted by Chris Hostetter <ho...@fucit.org>.
: Frustrated is the word :) I have looked at Solr...what I am worried
: about there is this: Solr says it requires an OS that supports hard
: links. Currently Windows does not to my knowledge. Someone seemed to
: make a comment that Windows could be supported...from what I know I
: don't think so. Not a deal breaker per say but then there is this: I
Solr does not require hardlinks, what the FAQ says is...
"The Replication features of Solr currently require an OS with the
ability to create hard links and rsync."
...which means if you want to use the replication system provided with
Solr as is, you need hardlinks and rsync. Solr is designed with the
replication as a very external portion of the system (it's just executing
shell calls specific in a config file) so it should be possible to plug in
a different replication system and use the existing hooks for generating
snapshoots on the master and loading snapshots on the slave ... it just
hasn't been a priority.
: have done a lot with the lucene API. I have created a custom query
: language to lucene query parser. I have changed the standard parser. I
: have made heavy use of Multi-Searchers. I am really tied into the Lucene
: API. I am worried about how easy it will be to integrate that into Solr.
Anything thing you do at "query time" with the Lucene API, can be done in
a SolrRequestHandler (which you write in Java and register in the solr
config file) -- change just a few method calls and you'll get a lot of
great caching features as well.
none of which really addresses the crux of your question....
: Can I index 30 million+ docs that range in size form 2-10kb on a single
: server in a Windows environment (accesss to a max of about 1.5 gig of
: RAM). The average search will need to be sorted by field not relevancy.
I can't say that I've personally built/used a lucene index of 30 million
docs, but i have talked to people who have done it .. they certainly had
some performance issues, but those issues were mainly related to the
volume of queries they got, not so much the size of their index. That
said: you are seriously hindering yourself with the Windows/RAM (The
FieldCache for your sort field (assuming it's an int) alone will be over
100MB), not to mention the fact that your index isn't static, so creating
a new searcher after you've made updates esentially halves the amount of
usable RAM you have to work with unless you're willing to close one
searcher before you're willing to open the new one.
I haven't played with Remote/Multi Searchers, but perhaps you should
open yourself up to the possibility of partitionining your index on
several boxes.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 30 milllion+ docs on a single server
Posted by Mark Miller <ma...@gmail.com>.
Frustrated is the word :) I have looked at Solr...what I am worried
about there is this: Solr says it requires an OS that supports hard
links. Currently Windows does not to my knowledge. Someone seemed to
make a comment that Windows could be supported...from what I know I
don't think so. Not a deal breaker per say but then there is this: I
have done a lot with the lucene API. I have created a custom query
language to lucene query parser. I have changed the standard parser. I
have made heavy use of Multi-Searchers. I am really tied into the Lucene
API. I am worried about how easy it will be to integrate that into Solr.
Perhaps I can just grab the distributed part of Solr but I do not know.
I have so much to do that worrying about a distributed search seems like
too big a scope for now. It seemed to me that breaking up the index with
an RMI searcher was the easiest approach anyway. In the end...I would
really like to stay on one server. This server will prob have multiple
procs...should I make sure I incorporate a parallel searcher option?
In the end I am really just hoping for some more insight into this exact
question:
Can I index 30 million+ docs that range in size form 2-10kb on a single
server in a Windows environment (accesss to a max of about 1.5 gig of
RAM). The average search will need to be sorted by field not relevancy.
Do you think its possible or a pipe dream? I realize I need to test to
find out...but I am looking for someone with experience to pipe in
before I get to that point.
Thanks for the response so far...I love the lucene mailing list.
Thanks,
Mark
Ray Tsang wrote:
> i've indexed 80m records and now up to 200m.. it can be done, and
> could've
> been done better. like the other said, architecture is important.
> have you
> considered looking into solr? i haven't kept up with it (and many of the
> mailing lists...), but looks very interesting.
>
> ray,
>
> On 8/12/06, Jason Polites <ja...@gmail.com> wrote:
>>
>> Sounds like you're a bit frustrated. Cheer up, the simple fact is that
>> engineering and business rarely see eye-to-eye. Just focus on the fact
>> that
>> what you have learnt from the process will help you, and they paid
>> for it
>> ;)
>>
>> On the issue at hand...Lucene should scale to this level, but you need a
>> good architecture behind it. Google has good indexing tech, but it's
>> their
>> architecture that allows them to spread the index across thousands of
>> servers which really gives it grunt (to the point that they invented
>> their
>> own RAID-style file system).
>>
>> Just think very carefully about the architecture underpinning the index.
>> Lucene is core-tech. It's up to you to provide the framework to make it
>> hum.
>>
>> On 8/12/06, Mark Miller <ma...@gmail.com> wrote:
>> >
>> > Tomi NA wrote:
>> > > On 8/12/06, Mark Miller <ma...@gmail.com> wrote:
>> > >> I've made a nice little archive application with lucene. I made
>> it to
>> > >> handle our largest need: 2.5 million docs or so on a single server.
>> Now
>> > >> the powers that be say: lets use it for a 30+ million document
>> archive
>> > >> on a single server! (each doc size maybe 10k max...as small as a
>> 1 or
>> > >> 2k) Please tell me why we are in trouble...please tell me why we
>> are
>> > >> not. I have tested up to 2 million docs without much trouble but 30
>> > >> million...the average search will include a sort on a field as
>> > >> well...can I search 30+ million docs with a sort? Man am I worried
>> > about
>> > >> that. Maybe the server will have 8 procs and 12 billion gigs of
>> RAM.
>> > >> Mabye. Even still, Tomcat seems to be able to launch with a max of
>> 1.5
>> > >> or 1.6 gig of Ram in Windows. What do you think? 30 million+ sounds
>> > like
>> > >> too much of a load to me for a single server. Not that they care
>> what
>> I
>> > >> think...I only wrote the thing (man I hate my job, offer me a
>> new one
>> > :)
>> > >> )...please...comments?
>> > >>
>> > >> Cheers,
>> > >>
>> > >> Miserable Mark
>> > >
>> > > I don't really understand what you're so worried about. Either it'll
>> > > work well with the setup you have, or it won't. It's really the size
>> > > of it. ;)
>> > > Seriously, you have a number of relatively cheap possibilities at
>> hand
>> > > to improve search performance: storing the index on a RAID 5 disk
>> > > array will let you read the indices very fast, using multicore CPUs,
>> > > adding memory and even if all that isn't good enough, you can always
>> > > use a small cluster (say, 4 nodes) of very, very inexpensive PCs
>> > > filled with a GB of RAM. You don't have to keep them inside the
>> > > regular UPS/backup/voult-protected area as the indices can always be
>> > > rebuilt (unlike e.g. data in transactional systems) and between 4 of
>> > > them they might cost like an entry-level server.
>> > > I'll let the experts speak now. :)
>> > >
>> > > t.n.a.
>> > >
>> > >
>> ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > > For additional commands, e-mail: java-user-help@lucene.apache.org
>> > >
>> > >
>> > Thanks for the tip...I am not too worried...I am miserable because I
>> > live in Dilbert land, not this particular incident. Spreading to
>> > multiple servers is a possibility but one I want to avoid...I wrote
>> this
>> > app on the side since our current product is crap...it still needs
>> a lot
>> > of work and thinking about distributing lucene at this point is a
>> little
>> > much...I never even have time to work on this project as it is
>> becuase I
>> > am currently tasked with porting the crap old project to Windows. I
>> need
>> > to do a bunch to shore up what I have. No one cares though...they
>> think
>> > that I have done nothing (or can't understand what I have done)
>> while at
>> > the same time they want to use what I havn't done to do what I made it
>> > for as well as this new super archive of 30 million + docs...in the
>> end
>> > I'll be looking for a new job...still curious about lucene scaling
>> to 30
>> > million docs with a sort on every search though (yes I know the
>> sort is
>> > cached...worries me too though...the sort will be on multiple and
>> > different fields depending no what the searcher wants...uggg...the
>> size
>> > of the caches....)
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>> >
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 30 milllion+ docs on a single server
Posted by Ray Tsang <sa...@gmail.com>.
i've indexed 80m records and now up to 200m.. it can be done, and could've
been done better. like the other said, architecture is important. have you
considered looking into solr? i haven't kept up with it (and many of the
mailing lists...), but looks very interesting.
ray,
On 8/12/06, Jason Polites <ja...@gmail.com> wrote:
>
> Sounds like you're a bit frustrated. Cheer up, the simple fact is that
> engineering and business rarely see eye-to-eye. Just focus on the fact
> that
> what you have learnt from the process will help you, and they paid for it
> ;)
>
> On the issue at hand...Lucene should scale to this level, but you need a
> good architecture behind it. Google has good indexing tech, but it's
> their
> architecture that allows them to spread the index across thousands of
> servers which really gives it grunt (to the point that they invented their
> own RAID-style file system).
>
> Just think very carefully about the architecture underpinning the index.
> Lucene is core-tech. It's up to you to provide the framework to make it
> hum.
>
> On 8/12/06, Mark Miller <ma...@gmail.com> wrote:
> >
> > Tomi NA wrote:
> > > On 8/12/06, Mark Miller <ma...@gmail.com> wrote:
> > >> I've made a nice little archive application with lucene. I made it to
> > >> handle our largest need: 2.5 million docs or so on a single server.
> Now
> > >> the powers that be say: lets use it for a 30+ million document
> archive
> > >> on a single server! (each doc size maybe 10k max...as small as a 1 or
> > >> 2k) Please tell me why we are in trouble...please tell me why we are
> > >> not. I have tested up to 2 million docs without much trouble but 30
> > >> million...the average search will include a sort on a field as
> > >> well...can I search 30+ million docs with a sort? Man am I worried
> > about
> > >> that. Maybe the server will have 8 procs and 12 billion gigs of RAM.
> > >> Mabye. Even still, Tomcat seems to be able to launch with a max of
> 1.5
> > >> or 1.6 gig of Ram in Windows. What do you think? 30 million+ sounds
> > like
> > >> too much of a load to me for a single server. Not that they care what
> I
> > >> think...I only wrote the thing (man I hate my job, offer me a new one
> > :)
> > >> )...please...comments?
> > >>
> > >> Cheers,
> > >>
> > >> Miserable Mark
> > >
> > > I don't really understand what you're so worried about. Either it'll
> > > work well with the setup you have, or it won't. It's really the size
> > > of it. ;)
> > > Seriously, you have a number of relatively cheap possibilities at hand
> > > to improve search performance: storing the index on a RAID 5 disk
> > > array will let you read the indices very fast, using multicore CPUs,
> > > adding memory and even if all that isn't good enough, you can always
> > > use a small cluster (say, 4 nodes) of very, very inexpensive PCs
> > > filled with a GB of RAM. You don't have to keep them inside the
> > > regular UPS/backup/voult-protected area as the indices can always be
> > > rebuilt (unlike e.g. data in transactional systems) and between 4 of
> > > them they might cost like an entry-level server.
> > > I'll let the experts speak now. :)
> > >
> > > t.n.a.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> > Thanks for the tip...I am not too worried...I am miserable because I
> > live in Dilbert land, not this particular incident. Spreading to
> > multiple servers is a possibility but one I want to avoid...I wrote this
> > app on the side since our current product is crap...it still needs a lot
> > of work and thinking about distributing lucene at this point is a little
> > much...I never even have time to work on this project as it is becuase I
> > am currently tasked with porting the crap old project to Windows. I need
> > to do a bunch to shore up what I have. No one cares though...they think
> > that I have done nothing (or can't understand what I have done) while at
> > the same time they want to use what I havn't done to do what I made it
> > for as well as this new super archive of 30 million + docs...in the end
> > I'll be looking for a new job...still curious about lucene scaling to 30
> > million docs with a sort on every search though (yes I know the sort is
> > cached...worries me too though...the sort will be on multiple and
> > different fields depending no what the searcher wants...uggg...the size
> > of the caches....)
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
Re: 30 milllion+ docs on a single server
Posted by Jason Polites <ja...@gmail.com>.
Sounds like you're a bit frustrated. Cheer up, the simple fact is that
engineering and business rarely see eye-to-eye. Just focus on the fact that
what you have learnt from the process will help you, and they paid for it ;)
On the issue at hand...Lucene should scale to this level, but you need a
good architecture behind it. Google has good indexing tech, but it's their
architecture that allows them to spread the index across thousands of
servers which really gives it grunt (to the point that they invented their
own RAID-style file system).
Just think very carefully about the architecture underpinning the index.
Lucene is core-tech. It's up to you to provide the framework to make it
hum.
On 8/12/06, Mark Miller <ma...@gmail.com> wrote:
>
> Tomi NA wrote:
> > On 8/12/06, Mark Miller <ma...@gmail.com> wrote:
> >> I've made a nice little archive application with lucene. I made it to
> >> handle our largest need: 2.5 million docs or so on a single server. Now
> >> the powers that be say: lets use it for a 30+ million document archive
> >> on a single server! (each doc size maybe 10k max...as small as a 1 or
> >> 2k) Please tell me why we are in trouble...please tell me why we are
> >> not. I have tested up to 2 million docs without much trouble but 30
> >> million...the average search will include a sort on a field as
> >> well...can I search 30+ million docs with a sort? Man am I worried
> about
> >> that. Maybe the server will have 8 procs and 12 billion gigs of RAM.
> >> Mabye. Even still, Tomcat seems to be able to launch with a max of 1.5
> >> or 1.6 gig of Ram in Windows. What do you think? 30 million+ sounds
> like
> >> too much of a load to me for a single server. Not that they care what I
> >> think...I only wrote the thing (man I hate my job, offer me a new one
> :)
> >> )...please...comments?
> >>
> >> Cheers,
> >>
> >> Miserable Mark
> >
> > I don't really understand what you're so worried about. Either it'll
> > work well with the setup you have, or it won't. It's really the size
> > of it. ;)
> > Seriously, you have a number of relatively cheap possibilities at hand
> > to improve search performance: storing the index on a RAID 5 disk
> > array will let you read the indices very fast, using multicore CPUs,
> > adding memory and even if all that isn't good enough, you can always
> > use a small cluster (say, 4 nodes) of very, very inexpensive PCs
> > filled with a GB of RAM. You don't have to keep them inside the
> > regular UPS/backup/voult-protected area as the indices can always be
> > rebuilt (unlike e.g. data in transactional systems) and between 4 of
> > them they might cost like an entry-level server.
> > I'll let the experts speak now. :)
> >
> > t.n.a.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> Thanks for the tip...I am not too worried...I am miserable because I
> live in Dilbert land, not this particular incident. Spreading to
> multiple servers is a possibility but one I want to avoid...I wrote this
> app on the side since our current product is crap...it still needs a lot
> of work and thinking about distributing lucene at this point is a little
> much...I never even have time to work on this project as it is becuase I
> am currently tasked with porting the crap old project to Windows. I need
> to do a bunch to shore up what I have. No one cares though...they think
> that I have done nothing (or can't understand what I have done) while at
> the same time they want to use what I havn't done to do what I made it
> for as well as this new super archive of 30 million + docs...in the end
> I'll be looking for a new job...still curious about lucene scaling to 30
> million docs with a sort on every search though (yes I know the sort is
> cached...worries me too though...the sort will be on multiple and
> different fields depending no what the searcher wants...uggg...the size
> of the caches....)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: 30 milllion+ docs on a single server
Posted by Mark Miller <ma...@gmail.com>.
Thanks for all of the useful info on this topic. You have been very
enlightening. My RAM requirements where obviously off the mark. Here is
my current understanding of this issue:
A standard 32-bit processor has access to 4GB of RAM. If your CPU
supports Physical Address Extension (PAE) the OS can access up to 64GB
of RAM, although a single application is still limited to 4GB. In
Windows, Address Windowing Extensions (AWE) solves this limitation but
you must use OS specific calls to manage your memory. Unix/Linux has
it's own memory mapping scheme that accomplishes something similar to AWE.
I have no problem requiring that the server i use be maxed to the
gills. I just want to support Windows as well as Unix/Linux/Sun. I was
shortsighted in my RAM comments.
As far as Solr: it looks like this will not necessarily solve my problem
of handling an index so big on a single server--the entire index is
replicated across each slave searcher--but if load is a factor
(according to Hoss, load will probably be the big issue, not the size of
the index) than this distribution is exactly what I need. I think that
the average load I can expect will be pretty low though. Updates to the
large index will also be relatively rare. 30-200 items added every
morning with the occasional update scattered throughout the day. There
will only be a few hundred searchers in total and they will not be
hammering the system but consulting it here and there throughout the day.
When I said that Solr required hardlinks I was referring to the
replication feature. My fault--replication was the benefit I was seeing
from solar and I was unclear. I bet solr's caching helps a lot too and I
will be looking into that. What I did not know was that Windows supports
hard links when using ntfs. Fsutil.exe creates them. This gives me hope
that cygwin might support them. I do not know if I can use the same cp
-l -r trick, but it at least it looks hopeful. I did not realize that
solr's replication feature was so external and easily changeable.
as far as rmi: splitting the index across several boxes seems relatively
easy (the index can be naturally partitioned without much trouble) but I
would still have to deal with sorting on a single box. I am not so
worried about that anymore though. Warming a new searcher and tons of
RAM should handle my sorts in an acceptable way.
It seems to me that one way or another I can make this happen.
I am wondering whether I should try and use a RAM directory. While I
am sure it might be faster, what happens when the power goes out? System
reboots? I would have to occasionally flush to disk anyway. Would it be
better to use a normal directory and turn the maxbuffered docs way up?
Thanks,
- Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: 30 milllion+ docs on a single server
Posted by Dejan Nenov <de...@jollyobject.com>.
The important detail here is what you mean by "single server"?
A high-end server will work just fine - you want 4GB+ or RAM and the fastest
disk/IO you can get; CPU speed is far less important; A nice Linux software
RAID and 5+ 15K SCSI disks will get you superb performance, at a reasonable
price.
-----Original Message-----
From: Mark Miller [mailto:markrmiller@gmail.com]
Sent: Friday, August 11, 2006 4:23 PM
To: java-user@lucene.apache.org
Subject: Re: 30 milllion+ docs on a single server
Tomi NA wrote:
> On 8/12/06, Mark Miller <ma...@gmail.com> wrote:
>> I've made a nice little archive application with lucene. I made it to
>> handle our largest need: 2.5 million docs or so on a single server. Now
>> the powers that be say: lets use it for a 30+ million document archive
>> on a single server! (each doc size maybe 10k max...as small as a 1 or
>> 2k) Please tell me why we are in trouble...please tell me why we are
>> not. I have tested up to 2 million docs without much trouble but 30
>> million...the average search will include a sort on a field as
>> well...can I search 30+ million docs with a sort? Man am I worried about
>> that. Maybe the server will have 8 procs and 12 billion gigs of RAM.
>> Mabye. Even still, Tomcat seems to be able to launch with a max of 1.5
>> or 1.6 gig of Ram in Windows. What do you think? 30 million+ sounds like
>> too much of a load to me for a single server. Not that they care what I
>> think...I only wrote the thing (man I hate my job, offer me a new one :)
>> )...please...comments?
>>
>> Cheers,
>>
>> Miserable Mark
>
> I don't really understand what you're so worried about. Either it'll
> work well with the setup you have, or it won't. It's really the size
> of it. ;)
> Seriously, you have a number of relatively cheap possibilities at hand
> to improve search performance: storing the index on a RAID 5 disk
> array will let you read the indices very fast, using multicore CPUs,
> adding memory and even if all that isn't good enough, you can always
> use a small cluster (say, 4 nodes) of very, very inexpensive PCs
> filled with a GB of RAM. You don't have to keep them inside the
> regular UPS/backup/voult-protected area as the indices can always be
> rebuilt (unlike e.g. data in transactional systems) and between 4 of
> them they might cost like an entry-level server.
> I'll let the experts speak now. :)
>
> t.n.a.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Thanks for the tip...I am not too worried...I am miserable because I
live in Dilbert land, not this particular incident. Spreading to
multiple servers is a possibility but one I want to avoid...I wrote this
app on the side since our current product is crap...it still needs a lot
of work and thinking about distributing lucene at this point is a little
much...I never even have time to work on this project as it is becuase I
am currently tasked with porting the crap old project to Windows. I need
to do a bunch to shore up what I have. No one cares though...they think
that I have done nothing (or can't understand what I have done) while at
the same time they want to use what I havn't done to do what I made it
for as well as this new super archive of 30 million + docs...in the end
I'll be looking for a new job...still curious about lucene scaling to 30
million docs with a sort on every search though (yes I know the sort is
cached...worries me too though...the sort will be on multiple and
different fields depending no what the searcher wants...uggg...the size
of the caches....)
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 30 milllion+ docs on a single server
Posted by Mark Miller <ma...@gmail.com>.
Tomi NA wrote:
> On 8/12/06, Mark Miller <ma...@gmail.com> wrote:
>> I've made a nice little archive application with lucene. I made it to
>> handle our largest need: 2.5 million docs or so on a single server. Now
>> the powers that be say: lets use it for a 30+ million document archive
>> on a single server! (each doc size maybe 10k max...as small as a 1 or
>> 2k) Please tell me why we are in trouble...please tell me why we are
>> not. I have tested up to 2 million docs without much trouble but 30
>> million...the average search will include a sort on a field as
>> well...can I search 30+ million docs with a sort? Man am I worried about
>> that. Maybe the server will have 8 procs and 12 billion gigs of RAM.
>> Mabye. Even still, Tomcat seems to be able to launch with a max of 1.5
>> or 1.6 gig of Ram in Windows. What do you think? 30 million+ sounds like
>> too much of a load to me for a single server. Not that they care what I
>> think...I only wrote the thing (man I hate my job, offer me a new one :)
>> )...please...comments?
>>
>> Cheers,
>>
>> Miserable Mark
>
> I don't really understand what you're so worried about. Either it'll
> work well with the setup you have, or it won't. It's really the size
> of it. ;)
> Seriously, you have a number of relatively cheap possibilities at hand
> to improve search performance: storing the index on a RAID 5 disk
> array will let you read the indices very fast, using multicore CPUs,
> adding memory and even if all that isn't good enough, you can always
> use a small cluster (say, 4 nodes) of very, very inexpensive PCs
> filled with a GB of RAM. You don't have to keep them inside the
> regular UPS/backup/voult-protected area as the indices can always be
> rebuilt (unlike e.g. data in transactional systems) and between 4 of
> them they might cost like an entry-level server.
> I'll let the experts speak now. :)
>
> t.n.a.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Thanks for the tip...I am not too worried...I am miserable because I
live in Dilbert land, not this particular incident. Spreading to
multiple servers is a possibility but one I want to avoid...I wrote this
app on the side since our current product is crap...it still needs a lot
of work and thinking about distributing lucene at this point is a little
much...I never even have time to work on this project as it is becuase I
am currently tasked with porting the crap old project to Windows. I need
to do a bunch to shore up what I have. No one cares though...they think
that I have done nothing (or can't understand what I have done) while at
the same time they want to use what I havn't done to do what I made it
for as well as this new super archive of 30 million + docs...in the end
I'll be looking for a new job...still curious about lucene scaling to 30
million docs with a sort on every search though (yes I know the sort is
cached...worries me too though...the sort will be on multiple and
different fields depending no what the searcher wants...uggg...the size
of the caches....)
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 30 milllion+ docs on a single server
Posted by Tomi NA <he...@gmail.com>.
On 8/12/06, Mark Miller <ma...@gmail.com> wrote:
> I've made a nice little archive application with lucene. I made it to
> handle our largest need: 2.5 million docs or so on a single server. Now
> the powers that be say: lets use it for a 30+ million document archive
> on a single server! (each doc size maybe 10k max...as small as a 1 or
> 2k) Please tell me why we are in trouble...please tell me why we are
> not. I have tested up to 2 million docs without much trouble but 30
> million...the average search will include a sort on a field as
> well...can I search 30+ million docs with a sort? Man am I worried about
> that. Maybe the server will have 8 procs and 12 billion gigs of RAM.
> Mabye. Even still, Tomcat seems to be able to launch with a max of 1.5
> or 1.6 gig of Ram in Windows. What do you think? 30 million+ sounds like
> too much of a load to me for a single server. Not that they care what I
> think...I only wrote the thing (man I hate my job, offer me a new one :)
> )...please...comments?
>
> Cheers,
>
> Miserable Mark
I don't really understand what you're so worried about. Either it'll
work well with the setup you have, or it won't. It's really the size
of it. ;)
Seriously, you have a number of relatively cheap possibilities at hand
to improve search performance: storing the index on a RAID 5 disk
array will let you read the indices very fast, using multicore CPUs,
adding memory and even if all that isn't good enough, you can always
use a small cluster (say, 4 nodes) of very, very inexpensive PCs
filled with a GB of RAM. You don't have to keep them inside the
regular UPS/backup/voult-protected area as the indices can always be
rebuilt (unlike e.g. data in transactional systems) and between 4 of
them they might cost like an entry-level server.
I'll let the experts speak now. :)
t.n.a.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re[2]: 30 milllion+ docs on a single server
Posted by Artem Vasiliev <ar...@gmail.com>.
Hi guys!
I have noticed many questions on the list vonsidering Lucene sorting
memory consumption and hope my solution can help someone.
I faced a memory/time consumption problem on sorting in Lucene back in
April. With a help of this list's experts I came to solution which I
like: documents from the sorting set (instead of given the field's
values from the whole index) are lazy-cached in a WeakHashMap so the
cached items are candidates for GC. I didn't create a patch yet (and
I'm going to make it) but the code is ready to be reused and it's used
for a while as a part of my open-source project, sharehound
(http://sharehound.sourceforge.net). The code can be now reached by a
CVS browser, it's in 4 classes in subdirectories of
http://sharehound.cvs.sourceforge.net/sharehound/jNetCrawler/src/java/org/apache/lucene/.
They (both classes, as a part of sharehound.jar, and sources) can also
be downloaded with the latest (1.1.3 alpha) sharehound release zip
file.
LazyCachingSortFactory class have an example of use in its header
comments, I duplicate it here:
/**
* Creates a Sort that doesn't use FieldCache and doesn't therefore load all the field values from the index while sorting.
* Instead it uses CachingIndexReader (via CachingDocFieldComparatorSource) to cache documents from which field values
* are fetched. If the caller Searcher is based on CachingIndexReader itself its document cache will be used here.
*
*
* An example of use:
...
hits = getQueryHits(query, getSort(listSorting.getSortFieldName(), listSorting.isSortDescending()));
...
public Sort getSort(String sortFieldName, boolean sortDescending) {
return LazyCachingSortFactory.create(sortFieldName, sortDescending);
}
...
protected Hits getQueryHits(Query query, Sort sort) throws IOException {
IndexSearcher indexSearcher = new IndexSearcher(CachingIndexReader.decorateIfNeeded(IndexReader.open(getIndexDir())));
return indexSearcher.search(query, sort);
}
*/
This solution does great work saving a memory and in case of
relatively search resultsets it's almost as fast as the default
implementation. Note that this solution is ready only for single-field
sorting currently.
Best regards,
Artem
OG> This is unlikely to work well/fast. It will depend on the
OG> size of the index (not in terms of the number of docs, but its
OG> physical size), the number of queries/second and desired query
OG> latency. If you can wait 10 seconds to get a query and if only a
OG> few queries are hitting the server at any one time, then you may
OG> be Ok. Having things be up to date with non-relevancy sorting
OG> will be quite tough. FieldCache will consume some RAM. Warming
OG> it up will take some number of seconds. Re-opening an
OG> IndexSearcher after index changes will also cost you a bit of time.
OG> Consider a 64-bit server with more RAM that allowed larger
OG> Java heaps, and try to fit your index into RAM.
OG> Otis
OG> ----- Original Message ----
OG> From: Mark Miller <ma...@gmail.com>
OG> To: java-user@lucene.apache.org
OG> Sent: Saturday, August 12, 2006 7:45:15 PM
OG> Subject: Re: 30 milllion+ docs on a single server
OG> The single server is important because I think it will take a lot of
OG> work to scale it to multiple servers. The index must allow for close to
OG> real-time updates and additions. It must also remain searchable at all
OG> times (other than than during the brief period of single updates and
OG> additions). If it is easy to scale this to multiple servers please tell
OG> me how.
OG> - Mark
>> Why is a single server so important? I can scale horizontally much
>> cheaper
>> than I scale vertically.
>>
>>
>>
>> On 8/11/06, Mark Miller <ma...@gmail.com> wrote:
>>>
>>> I've made a nice little archive application with lucene. I made it to
>>> handle our largest need: 2.5 million docs or so on a single server. Now
>>> the powers that be say: lets use it for a 30+ million document archive
>>> on a single server! (each doc size maybe 10k max...as small as a 1 or
>>> 2k) Please tell me why we are in trouble...please tell me why we are
>>> not. I have tested up to 2 million docs without much trouble but 30
>>> million...the average search will include a sort on a field as
>>> well...can I search 30+ million docs with a sort? Man am I worried about
>>> that. Maybe the server will have 8 procs and 12 billion gigs of RAM.
>>> Mabye. Even still, Tomcat seems to be able to launch with a max of 1.5
>>> or 1.6 gig of Ram in Windows. What do you think? 30 million+ sounds like
>>> too much of a load to me for a single server. Not that they care what I
>>> think...I only wrote the thing (man I hate my job, offer me a new one :)
>>> )...please...comments?
>>>
>>> Cheers,
>>>
>>> Miserable Mark
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
OG> ---------------------------------------------------------------------
OG> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
OG> For additional commands, e-mail: java-user-help@lucene.apache.org
OG> ---------------------------------------------------------------------
OG> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
OG> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 30 milllion+ docs on a single server
Posted by Jeff Rodenburg <je...@gmail.com>.
On 8/12/06, Mark Miller <ma...@gmail.com> wrote:
>
> The single server is important because I think it will take a lot of
> work to scale it to multiple servers. The index must allow for close to
> real-time updates and additions. It must also remain searchable at all
> times (other than than during the brief period of single updates and
> additions). If it is easy to scale this to multiple servers please tell
> me how.
>
It can take quite a bit of work to implement a multiple-server index system;
we did it last year, building an operational wrapper around Lucene. Wish
Solr had been around then. ;-)
I've done both the Windows and the Linux route. Windows certainly comes
from a scale-up mentality, though we made it work in a scale-out model. Our
requirements were the same as yours: near real-time updates & additions,
always-on searchability, etc. It takes work, but it can be done. We're
serving searches across 6 different types of indexes, with the indexes
spread across the server farm (no single server has the full composite
index). Our search availability for this year is damn near 5 nines. If you
haven't looked at Windows 64-bit, let me save you some time. You don't gain
as much as you might expect; the point of diminishing returns appears to
have certainly been met with Windows Server. We'll apply a similar strategy
to Solr, in that we'll likely run Solr clusters for our composite index.
The best way to explain "how" is to simply refer you to Solr, from an
operational perspective. The only thing that Solr doesn't have that we do
is rolling together results from multiple searchers, and that's simply an
out-of-the-box configuration; it's not a major ordeal to change that to meet
our needs.
Hope this helps.
-- j
Re: 30 milllion+ docs on a single server
Posted by Otis Gospodnetic <ot...@yahoo.com>.
This is unlikely to work well/fast. It will depend on the size of the index (not in terms of the number of docs, but its physical size), the number of queries/second and desired query latency. If you can wait 10 seconds to get a query and if only a few queries are hitting the server at any one time, then you may be Ok. Having things be up to date with non-relevancy sorting will be quite tough. FieldCache will consume some RAM. Warming it up will take some number of seconds. Re-opening an IndexSearcher after index changes will also cost you a bit of time.
Consider a 64-bit server with more RAM that allowed larger Java heaps, and try to fit your index into RAM.
Otis
----- Original Message ----
From: Mark Miller <ma...@gmail.com>
To: java-user@lucene.apache.org
Sent: Saturday, August 12, 2006 7:45:15 PM
Subject: Re: 30 milllion+ docs on a single server
The single server is important because I think it will take a lot of
work to scale it to multiple servers. The index must allow for close to
real-time updates and additions. It must also remain searchable at all
times (other than than during the brief period of single updates and
additions). If it is easy to scale this to multiple servers please tell
me how.
- Mark
> Why is a single server so important? I can scale horizontally much
> cheaper
> than I scale vertically.
>
>
>
> On 8/11/06, Mark Miller <ma...@gmail.com> wrote:
>>
>> I've made a nice little archive application with lucene. I made it to
>> handle our largest need: 2.5 million docs or so on a single server. Now
>> the powers that be say: lets use it for a 30+ million document archive
>> on a single server! (each doc size maybe 10k max...as small as a 1 or
>> 2k) Please tell me why we are in trouble...please tell me why we are
>> not. I have tested up to 2 million docs without much trouble but 30
>> million...the average search will include a sort on a field as
>> well...can I search 30+ million docs with a sort? Man am I worried about
>> that. Maybe the server will have 8 procs and 12 billion gigs of RAM.
>> Mabye. Even still, Tomcat seems to be able to launch with a max of 1.5
>> or 1.6 gig of Ram in Windows. What do you think? 30 million+ sounds like
>> too much of a load to me for a single server. Not that they care what I
>> think...I only wrote the thing (man I hate my job, offer me a new one :)
>> )...please...comments?
>>
>> Cheers,
>>
>> Miserable Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 30 milllion+ docs on a single server
Posted by Mark Miller <ma...@gmail.com>.
The single server is important because I think it will take a lot of
work to scale it to multiple servers. The index must allow for close to
real-time updates and additions. It must also remain searchable at all
times (other than than during the brief period of single updates and
additions). If it is easy to scale this to multiple servers please tell
me how.
- Mark
> Why is a single server so important? I can scale horizontally much
> cheaper
> than I scale vertically.
>
>
>
> On 8/11/06, Mark Miller <ma...@gmail.com> wrote:
>>
>> I've made a nice little archive application with lucene. I made it to
>> handle our largest need: 2.5 million docs or so on a single server. Now
>> the powers that be say: lets use it for a 30+ million document archive
>> on a single server! (each doc size maybe 10k max...as small as a 1 or
>> 2k) Please tell me why we are in trouble...please tell me why we are
>> not. I have tested up to 2 million docs without much trouble but 30
>> million...the average search will include a sort on a field as
>> well...can I search 30+ million docs with a sort? Man am I worried about
>> that. Maybe the server will have 8 procs and 12 billion gigs of RAM.
>> Mabye. Even still, Tomcat seems to be able to launch with a max of 1.5
>> or 1.6 gig of Ram in Windows. What do you think? 30 million+ sounds like
>> too much of a load to me for a single server. Not that they care what I
>> think...I only wrote the thing (man I hate my job, offer me a new one :)
>> )...please...comments?
>>
>> Cheers,
>>
>> Miserable Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 30 milllion+ docs on a single server
Posted by Jeff Rodenburg <je...@gmail.com>.
Why is a single server so important? I can scale horizontally much cheaper
than I scale vertically.
On 8/11/06, Mark Miller <ma...@gmail.com> wrote:
>
> I've made a nice little archive application with lucene. I made it to
> handle our largest need: 2.5 million docs or so on a single server. Now
> the powers that be say: lets use it for a 30+ million document archive
> on a single server! (each doc size maybe 10k max...as small as a 1 or
> 2k) Please tell me why we are in trouble...please tell me why we are
> not. I have tested up to 2 million docs without much trouble but 30
> million...the average search will include a sort on a field as
> well...can I search 30+ million docs with a sort? Man am I worried about
> that. Maybe the server will have 8 procs and 12 billion gigs of RAM.
> Mabye. Even still, Tomcat seems to be able to launch with a max of 1.5
> or 1.6 gig of Ram in Windows. What do you think? 30 million+ sounds like
> too much of a load to me for a single server. Not that they care what I
> think...I only wrote the thing (man I hate my job, offer me a new one :)
> )...please...comments?
>
> Cheers,
>
> Miserable Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>