You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Christian von Wendt-Jensen <Ch...@infopaq.com> on 2013/03/15 09:49:42 UTC
Storage solutions recommendations
Hi,
I want to find what your experiences are with different storage setups.
We tried running a master/slave setup on the SAN but quickly realized that the master did not index fast enough. We didn't run with soft commit though – maybe that would change the conclusion?
The slaves seemed to run OK with data on the SAN, but as soon as replication was enabled, it died. Replication took hours and drained resources preventing good performance on the replicas. The cache warmup time took forever.
How does YOUR setup look like, and what storage solutions could YOU recommend? SAN? Local disc? Local SSD? Softcommit?
Med venlig hilsen / Best Regards
Christian von Wendt-Jensen
IT Team Lead, Customer Solutions
Infopaq International A/S
Kgs. Nytorv 22
DK-1050 København K
Phone +45 36 99 00 00
Mobile +45 31 17 10 07
Email christian.sonne.jensen@infopaq.com<ma...@infopaq.com>
Web www.infopaq.com<http://www.infopaq.com/>
DISCLAIMER:
This e-mail and accompanying documents contain privileged confidential information. The information is intended only for the recipient(s) named. Any unauthorised disclosure, copying, distribution, exploitation or the taking of any action in reliance of the content of this e-mail is strictly prohibited. If you have received this e-mail in error we would be obliged if you would delete the e-mail and attachments and notify the dispatcher by return e-mail or at +45 36 99 00 00
P Please consider the environment before printing this mail note.
Re: Storage solutions recommendations
Posted by Julián Arocena <ja...@temperies.com>.
Hi,
one of our clients provides for an important Argentine telco, a complete
system to integrate and organize in a simple system large volumes of data
with information about customers,
transactions, security risk, potential frauds among other activities all in
real time. For text searching they use Scotas OLS (www.scotas.com/products),
that is a native integration of Solr into Oracle dmbs.
OLS replaces the Lucene inverted index storage, which by default is stored
on the OS file-system, by Oracle Secure File BLOBs, resulting in high
scalable, secure and transactional storage. A summarizes advantages of this
approach are:
- Transactional storage, a parallel process can do insert or optimize
operations and if they fail simply do a rollback and nothing happens to
other concurrent sessions.
- Compression and encryption using Secure File functionality, applicable
to Lucene Inverted Index storage and Solr configuration files.
- Shared storage for Lucene Inverted Index, on RAC installations several
processes across nodes can use the storage transparently.
I hope this information can be useful for you.
Bye,
Julian
2013/3/15 Otis Gospodnetic <ot...@gmail.com>
> Hi,
>
> Most of our clients/customers use local storage. Some use SSDs and some
> SANs, and those with extra cash use SANs with SSDs.
>
> But what you wrote needs more detail because sources of poor performance
> can come from many places and there are a lot or very different setups out
> there that work in one situation but not in another.
>
> Otis
> Solr & ElasticSearch Support
> http://sematext.com/
> On Mar 15, 2013 6:37 AM, "Christian von Wendt-Jensen" <
> Christian.vonWendt-Jensen@infopaq.com> wrote:
>
> > Hi,
> >
> > I want to find what your experiences are with different storage setups.
> >
> > We tried running a master/slave setup on the SAN but quickly realized
> that
> > the master did not index fast enough. We didn't run with soft commit
> though
> > – maybe that would change the conclusion?
> > The slaves seemed to run OK with data on the SAN, but as soon as
> > replication was enabled, it died. Replication took hours and drained
> > resources preventing good performance on the replicas. The cache warmup
> > time took forever.
> >
> > How does YOUR setup look like, and what storage solutions could YOU
> > recommend? SAN? Local disc? Local SSD? Softcommit?
> >
> >
> >
> >
> >
> > Med venlig hilsen / Best Regards
> >
> > Christian von Wendt-Jensen
> > IT Team Lead, Customer Solutions
> >
> > Infopaq International A/S
> > Kgs. Nytorv 22
> > DK-1050 København K
> >
> > Phone +45 36 99 00 00
> > Mobile +45 31 17 10 07
> > Email christian.sonne.jensen@infopaq.com<mailto:
> > christian.sonne.jensen@infopaq.com>
> > Web www.infopaq.com<http://www.infopaq.com/>
> >
> >
> >
> >
> >
> >
> >
> >
> > DISCLAIMER:
> > This e-mail and accompanying documents contain privileged confidential
> > information. The information is intended only for the recipient(s) named.
> > Any unauthorised disclosure, copying, distribution, exploitation or the
> > taking of any action in reliance of the content of this e-mail is
> strictly
> > prohibited. If you have received this e-mail in error we would be obliged
> > if you would delete the e-mail and attachments and notify the dispatcher
> by
> > return e-mail or at +45 36 99 00 00
> > P Please consider the environment before printing this mail note.
> >
>
Re: Storage solutions recommendations
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
Most of our clients/customers use local storage. Some use SSDs and some
SANs, and those with extra cash use SANs with SSDs.
But what you wrote needs more detail because sources of poor performance
can come from many places and there are a lot or very different setups out
there that work in one situation but not in another.
Otis
Solr & ElasticSearch Support
http://sematext.com/
On Mar 15, 2013 6:37 AM, "Christian von Wendt-Jensen" <
Christian.vonWendt-Jensen@infopaq.com> wrote:
> Hi,
>
> I want to find what your experiences are with different storage setups.
>
> We tried running a master/slave setup on the SAN but quickly realized that
> the master did not index fast enough. We didn't run with soft commit though
> – maybe that would change the conclusion?
> The slaves seemed to run OK with data on the SAN, but as soon as
> replication was enabled, it died. Replication took hours and drained
> resources preventing good performance on the replicas. The cache warmup
> time took forever.
>
> How does YOUR setup look like, and what storage solutions could YOU
> recommend? SAN? Local disc? Local SSD? Softcommit?
>
>
>
>
>
> Med venlig hilsen / Best Regards
>
> Christian von Wendt-Jensen
> IT Team Lead, Customer Solutions
>
> Infopaq International A/S
> Kgs. Nytorv 22
> DK-1050 København K
>
> Phone +45 36 99 00 00
> Mobile +45 31 17 10 07
> Email christian.sonne.jensen@infopaq.com<mailto:
> christian.sonne.jensen@infopaq.com>
> Web www.infopaq.com<http://www.infopaq.com/>
>
>
>
>
>
>
>
>
> DISCLAIMER:
> This e-mail and accompanying documents contain privileged confidential
> information. The information is intended only for the recipient(s) named.
> Any unauthorised disclosure, copying, distribution, exploitation or the
> taking of any action in reliance of the content of this e-mail is strictly
> prohibited. If you have received this e-mail in error we would be obliged
> if you would delete the e-mail and attachments and notify the dispatcher by
> return e-mail or at +45 36 99 00 00
> P Please consider the environment before printing this mail note.
>
Re: Storage solutions recommendations
Posted by Erick Erickson <er...@gmail.com>.
bq: the master did not index fast enough
Stop. Pause, analyze <G>.....
Maybe you're already done this, but have you identified _why_ the master
doesn't "index fast enough"? If you're indexing from SolrJ, try commenting
out _just_ the line server.add(doc/doclist). I can't tell you how many
setups I've seen where the problem isn't Solr, but acquiring the data from
the system of record.
What rates do you require? What rates are you seeing? Perhaps with a bit
more information people could provide more insightful answers. Your
question has potential to be an XY problem, see:
http://www.perlmonks.org/index.pl?node_id=542341
Best
Erick
On Fri, Mar 15, 2013 at 4:49 AM, Christian von Wendt-Jensen <
Christian.vonWendt-Jensen@infopaq.com> wrote:
> Hi,
>
> I want to find what your experiences are with different storage setups.
>
> We tried running a master/slave setup on the SAN but quickly realized that
> the master did not index fast enough. We didn't run with soft commit though
> – maybe that would change the conclusion?
> The slaves seemed to run OK with data on the SAN, but as soon as
> replication was enabled, it died. Replication took hours and drained
> resources preventing good performance on the replicas. The cache warmup
> time took forever.
>
> How does YOUR setup look like, and what storage solutions could YOU
> recommend? SAN? Local disc? Local SSD? Softcommit?
>
>
>
>
>
> Med venlig hilsen / Best Regards
>
> Christian von Wendt-Jensen
> IT Team Lead, Customer Solutions
>
> Infopaq International A/S
> Kgs. Nytorv 22
> DK-1050 København K
>
> Phone +45 36 99 00 00
> Mobile +45 31 17 10 07
> Email christian.sonne.jensen@infopaq.com<mailto:
> christian.sonne.jensen@infopaq.com>
> Web www.infopaq.com<http://www.infopaq.com/>
>
>
>
>
>
>
>
>
> DISCLAIMER:
> This e-mail and accompanying documents contain privileged confidential
> information. The information is intended only for the recipient(s) named.
> Any unauthorised disclosure, copying, distribution, exploitation or the
> taking of any action in reliance of the content of this e-mail is strictly
> prohibited. If you have received this e-mail in error we would be obliged
> if you would delete the e-mail and attachments and notify the dispatcher by
> return e-mail or at +45 36 99 00 00
> P Please consider the environment before printing this mail note.
>