You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Matthias Hess <ma...@bluewin.ch> on 2009/09/26 09:36:23 UTC
Storing a Lucene Index on a SAN Storage: good idea?
Hello
We are currently implementing our first Lucene project. We are building
an application which will index public Records on the internet, about
200'000 documents, each document is about 150 k in size. Our customer
would like to store the Lucene index on a SAN disk.
We recommended the use of high speed local disks, but our customer would
prefer SAN for their better managability.
Does anybody have good or bad experiences with SAN disks?
Are there parameters like #read operations, data read rate or whatever,
which must be met to have a performance which rivals a good "local disk"?
Thanks for sharing your ideas and opinions!
Kind Regards
Matthias Hess
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Storing a Lucene Index on a SAN Storage: good idea?
Posted by Petite Abeille <pe...@mac.com>.
On Sep 26, 2009, at 9:36 AM, Matthias Hess wrote:
> Does anybody have good or bad experiences with SAN disks?
Yea, make sure that your "high end" SAN doesn't end up storing your
files on one physical disk :P
In short, a SAN can be fiendishly difficult to setup properly,
specially if it has to be shared among different applications with
different needs and wants. Too much of a good thing in general. Not
worth the price, the complexity, nor the pain.
A decent alternative to a SAN would be something like a couple of
dedicated Sun Thor (aka Sun Fire X4540 Server):
http://www.sun.com/servers/x64/x4540/
This gives you 48 disks to transparently spread you load across in a
rather compact, simple package. Very handy for IO intensive systems.
As always, YMMV.
Cheers,
--
PA.
http://alt.textdrive.com/nanoki/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Storing a Lucene Index on a SAN Storage: good idea?
Posted by Uwe Schindler <uw...@thetaphi.de>.
If it's only 200,000 documents, I think the index will not be very big (a
few Gigabytes), especially, if the documents indexed are 150 kiB raw size
documents like PDFs or DOCs that have much binary overhead not indexed. Most
times the number of distinct terms is low. Index size is most times a lot
smaller than the original documents. If you also store the original doc
contents in the index, it get's bigger, but the time to access the same like
if they are normal files on the SAN.
This config would also run on my laptop :-) I think, the index will
completely be in memory after some time and so there is no difference
between SAN/local HDDs.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Matthias Hess [mailto:mat.hess@bluewin.ch]
> Sent: Saturday, September 26, 2009 9:36 AM
> To: java-user@lucene.apache.org
> Subject: Storing a Lucene Index on a SAN Storage: good idea?
>
> Hello
>
> We are currently implementing our first Lucene project. We are building
> an application which will index public Records on the internet, about
> 200'000 documents, each document is about 150 k in size. Our customer
> would like to store the Lucene index on a SAN disk.
> We recommended the use of high speed local disks, but our customer would
> prefer SAN for their better managability.
>
> Does anybody have good or bad experiences with SAN disks?
> Are there parameters like #read operations, data read rate or whatever,
> which must be met to have a performance which rivals a good "local disk"?
>
> Thanks for sharing your ideas and opinions!
>
> Kind Regards
> Matthias Hess
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org