You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Deepak Konidena <de...@gmail.com> on 2013/09/11 20:35:27 UTC
Distributing lucene segments across multiple disks.
Hi,
I know that SolrCloud allows you to have multiple shards on different
machines (or a single machine). But it requires a zookeeper installation
for doing things like leader election, leader availability, etc
While SolrCloud may be the ideal solution for my usecase eventually, I'd
like to know if there's a way I can point my Solr instance to read lucene
segments distributed across different disks attached to the same machine.
Thanks!
-Deepak
RE: Distributing lucene segments across multiple disks.
Posted by Greg Walters <gw...@sherpaanalytics.com>.
Deepak,
Sorry for not being more verbose in my previous suggestion. As I take your question, you'd like to spread your index files across multiple disks (for performance or space reasons I assume). If you used even a basic md-raid setup you could then format the raid device and thus your entire set of disks with your favorite filesystem, mount it in one directory in the directory tree then configure it as the data directory that solr uses.
This setup would accomplish your goal of having the lucene indexes spread across multiple disks without the complexity of using multiple solr cores/collections.
Thanks,
Greg
-----Original Message-----
From: Deepak Konidena [mailto:deepakkoni@gmail.com]
Sent: Wednesday, September 11, 2013 2:26 PM
To: solr-user@lucene.apache.org
Subject: Re: Distributing lucene segments across multiple disks.
@Greg - Are you suggesting RAID as a replacement for Solr or making Solr work with RAID? Could you elaborate more on the latter, if that's you meant?
We make use of solr's advanced text processing features which would be hard to replicate just using RAID.
-Deepak
On Wed, Sep 11, 2013 at 12:11 PM, Greg Walters <gwalters@sherpaanalytics.com
> wrote:
> Why not use some form of RAID for your index store? You'd get the
> performance benefit of multiple disks without the complexity of
> managing them via solr.
>
> Thanks,
> Greg
>
>
>
> -----Original Message-----
> From: Deepak Konidena [mailto:deepakkoni@gmail.com]
> Sent: Wednesday, September 11, 2013 2:07 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Distributing lucene segments across multiple disks.
>
> Are you suggesting a multi-core setup, where all the cores share the
> same schema, and the cores lie on different disks?
>
> Basically, I'd like to know if I can distribute shards/segments on a
> single machine (with multiple disks) without the use of zookeeper.
>
>
>
>
>
> -Deepak
>
>
>
> On Wed, Sep 11, 2013 at 11:55 AM, Upayavira <uv...@odoko.co.uk> wrote:
>
> > I think you'll find it hard to distribute different segments between
> > disks, as they are typically stored in the same directory.
> >
> > However, instantiating separate cores on different disks should be
> > straight-forward enough, and would give you a performance benefit.
> >
> > I've certainly heard of that done at Amazon, with a separate EBS
> > volume per core giving some performance improvement.
> >
> > Upayavira
> >
> > On Wed, Sep 11, 2013, at 07:35 PM, Deepak Konidena wrote:
> > > Hi,
> > >
> > > I know that SolrCloud allows you to have multiple shards on
> > > different machines (or a single machine). But it requires a
> > > zookeeper installation for doing things like leader election,
> > > leader availability, etc
> > >
> > > While SolrCloud may be the ideal solution for my usecase
> > > eventually, I'd like to know if there's a way I can point my Solr
> > > instance to read lucene segments distributed across different
> > > disks attached to
> the same machine.
> > >
> > > Thanks!
> > >
> > > -Deepak
> >
>
Re: Distributing lucene segments across multiple disks.
Posted by Deepak Konidena <de...@gmail.com>.
@Greg - Are you suggesting RAID as a replacement for Solr or making Solr
work with RAID? Could you elaborate more on the latter, if that's you
meant?
We make use of solr's advanced text processing features which would be hard
to replicate just using RAID.
-Deepak
On Wed, Sep 11, 2013 at 12:11 PM, Greg Walters <gwalters@sherpaanalytics.com
> wrote:
> Why not use some form of RAID for your index store? You'd get the
> performance benefit of multiple disks without the complexity of managing
> them via solr.
>
> Thanks,
> Greg
>
>
>
> -----Original Message-----
> From: Deepak Konidena [mailto:deepakkoni@gmail.com]
> Sent: Wednesday, September 11, 2013 2:07 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Distributing lucene segments across multiple disks.
>
> Are you suggesting a multi-core setup, where all the cores share the same
> schema, and the cores lie on different disks?
>
> Basically, I'd like to know if I can distribute shards/segments on a
> single machine (with multiple disks) without the use of zookeeper.
>
>
>
>
>
> -Deepak
>
>
>
> On Wed, Sep 11, 2013 at 11:55 AM, Upayavira <uv...@odoko.co.uk> wrote:
>
> > I think you'll find it hard to distribute different segments between
> > disks, as they are typically stored in the same directory.
> >
> > However, instantiating separate cores on different disks should be
> > straight-forward enough, and would give you a performance benefit.
> >
> > I've certainly heard of that done at Amazon, with a separate EBS
> > volume per core giving some performance improvement.
> >
> > Upayavira
> >
> > On Wed, Sep 11, 2013, at 07:35 PM, Deepak Konidena wrote:
> > > Hi,
> > >
> > > I know that SolrCloud allows you to have multiple shards on
> > > different machines (or a single machine). But it requires a
> > > zookeeper installation for doing things like leader election, leader
> > > availability, etc
> > >
> > > While SolrCloud may be the ideal solution for my usecase eventually,
> > > I'd like to know if there's a way I can point my Solr instance to
> > > read lucene segments distributed across different disks attached to
> the same machine.
> > >
> > > Thanks!
> > >
> > > -Deepak
> >
>
RE: Distributing lucene segments across multiple disks.
Posted by Greg Walters <gw...@sherpaanalytics.com>.
Why not use some form of RAID for your index store? You'd get the performance benefit of multiple disks without the complexity of managing them via solr.
Thanks,
Greg
-----Original Message-----
From: Deepak Konidena [mailto:deepakkoni@gmail.com]
Sent: Wednesday, September 11, 2013 2:07 PM
To: solr-user@lucene.apache.org
Subject: Re: Distributing lucene segments across multiple disks.
Are you suggesting a multi-core setup, where all the cores share the same schema, and the cores lie on different disks?
Basically, I'd like to know if I can distribute shards/segments on a single machine (with multiple disks) without the use of zookeeper.
-Deepak
On Wed, Sep 11, 2013 at 11:55 AM, Upayavira <uv...@odoko.co.uk> wrote:
> I think you'll find it hard to distribute different segments between
> disks, as they are typically stored in the same directory.
>
> However, instantiating separate cores on different disks should be
> straight-forward enough, and would give you a performance benefit.
>
> I've certainly heard of that done at Amazon, with a separate EBS
> volume per core giving some performance improvement.
>
> Upayavira
>
> On Wed, Sep 11, 2013, at 07:35 PM, Deepak Konidena wrote:
> > Hi,
> >
> > I know that SolrCloud allows you to have multiple shards on
> > different machines (or a single machine). But it requires a
> > zookeeper installation for doing things like leader election, leader
> > availability, etc
> >
> > While SolrCloud may be the ideal solution for my usecase eventually,
> > I'd like to know if there's a way I can point my Solr instance to
> > read lucene segments distributed across different disks attached to the same machine.
> >
> > Thanks!
> >
> > -Deepak
>
Re: Distributing lucene segments across multiple disks.
Posted by Deepak Konidena <de...@gmail.com>.
Very helpful link. Thanks for sharing that.
-Deepak
On Wed, Sep 11, 2013 at 4:34 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 9/11/2013 4:16 PM, Deepak Konidena wrote:
>
>> As far as RAM usage goes, I believe we set the heap size to about 40% of
>> the RAM and less than 10% is available for OS caching ( since replica
>> takes
>> another 40%). Why does unallocated RAM help? How does it impact
>> performance
>> under load?
>>
>
> Because once the data is in the OS disk cache, reading it becomes
> instantaneous, it doesn't need to go out to the disk. Disks are glacial
> compared to RAM. Even SSD has a far slower response time. Any recent
> operating system does this automatically, including the one from Redmond
> that we all love to hate.
>
> http://blog.thetaphi.de/2012/**07/use-lucenes-mmapdirectory-**
> on-64bit.html<http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html>
>
> Thanks,
> Shawn
>
>
Re: Distributing lucene segments across multiple disks.
Posted by Shawn Heisey <so...@elyograg.org>.
On 9/11/2013 4:16 PM, Deepak Konidena wrote:
> As far as RAM usage goes, I believe we set the heap size to about 40% of
> the RAM and less than 10% is available for OS caching ( since replica takes
> another 40%). Why does unallocated RAM help? How does it impact performance
> under load?
Because once the data is in the OS disk cache, reading it becomes
instantaneous, it doesn't need to go out to the disk. Disks are glacial
compared to RAM. Even SSD has a far slower response time. Any recent
operating system does this automatically, including the one from Redmond
that we all love to hate.
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Thanks,
Shawn
Re: Distributing lucene segments across multiple disks.
Posted by Deepak Konidena <de...@gmail.com>.
@Greg - Thanks for the suggestion. Will pass it along to my folks.
@Shawn - That's the link I was looking for 'non-SolrCloud approach to
distributed search'. Thanks for passing that along. Will give it a try.
As far as RAM usage goes, I believe we set the heap size to about 40% of
the RAM and less than 10% is available for OS caching ( since replica takes
another 40%). Why does unallocated RAM help? How does it impact performance
under load?
-Deepak
On Wed, Sep 11, 2013 at 2:50 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 9/11/2013 2:57 PM, Deepak Konidena wrote:
>
>> I guess at this point in the discussion, I should probably give some more
>> background on why I am doing what I am doing. Having a single Solr shard
>> (multiple segments) on the same disk is posing severe performance problems
>> under load,in that, calls to Solr cause a lot of connection timeouts. When
>> we looked at the ganglia stats for the Solr box, we saw that while memory,
>> cpu and network usage were quite normal, the i/o wait spiked. We are
>> unsure
>> on what caused the i/o wait and why there were no spikes in the cpu/memory
>> usage. Since the Solr box is a beefy box (multi-core setup, huge ram,
>> SSD),
>> we'd like to distribute the segments to multiple locations (disks) and see
>> whether this improves performance under load.
>>
>> @Greg - Thanks for clarifying that. I just learnt that I can't set them
>> up
>> using RAID as some of them are SSDs and some others are SATA (spinning
>> disks).
>>
>> @Shawn Heisey - Could you elaborate more about the "broker" core and
>> delegating the requests to other cores?
>>
>
> On the broker core - I have a core on my servers that has no index of its
> own. In the /select handler (and others) I have placed a shards parameter,
> and many of them also have a shards.qt parameter. The shards paramter is
> how a non-cloud distributed search is done.
>
> http://wiki.apache.org/solr/**DistributedSearch<http://wiki.apache.org/solr/DistributedSearch>
>
> Addressing your first paragraph: You say that you have lots of RAM ... but
> is there a lot of unallocated RAM that the OS can use for caching, or is it
> mostly allocated to processes, such as the java heap for Solr?
>
> Depending on exactly how your indexes are composed, you need up to 100% of
> the total index size available as unallocated RAM. With SSD, the
> requirement is less, but cannot be ignored. I personally wouldn't go below
> about 25-50% even with SSD, and I'd plan on 50-100% for regular disks.
>
> There is some evidence to suggest that you only need unallocated RAM equal
> to 10% of your index size for caching with SSD, but that is only likely to
> work if you have a lot of stored (as opposed to indexed) data. If most of
> your index is unstored, then more would be required.
>
> Thanks,
> Shawn
>
>
Re: Distributing lucene segments across multiple disks.
Posted by Shawn Heisey <so...@elyograg.org>.
On 9/11/2013 2:57 PM, Deepak Konidena wrote:
> I guess at this point in the discussion, I should probably give some more
> background on why I am doing what I am doing. Having a single Solr shard
> (multiple segments) on the same disk is posing severe performance problems
> under load,in that, calls to Solr cause a lot of connection timeouts. When
> we looked at the ganglia stats for the Solr box, we saw that while memory,
> cpu and network usage were quite normal, the i/o wait spiked. We are unsure
> on what caused the i/o wait and why there were no spikes in the cpu/memory
> usage. Since the Solr box is a beefy box (multi-core setup, huge ram, SSD),
> we'd like to distribute the segments to multiple locations (disks) and see
> whether this improves performance under load.
>
> @Greg - Thanks for clarifying that. I just learnt that I can't set them up
> using RAID as some of them are SSDs and some others are SATA (spinning
> disks).
>
> @Shawn Heisey - Could you elaborate more about the "broker" core and
> delegating the requests to other cores?
On the broker core - I have a core on my servers that has no index of
its own. In the /select handler (and others) I have placed a shards
parameter, and many of them also have a shards.qt parameter. The shards
paramter is how a non-cloud distributed search is done.
http://wiki.apache.org/solr/DistributedSearch
Addressing your first paragraph: You say that you have lots of RAM ...
but is there a lot of unallocated RAM that the OS can use for caching,
or is it mostly allocated to processes, such as the java heap for Solr?
Depending on exactly how your indexes are composed, you need up to 100%
of the total index size available as unallocated RAM. With SSD, the
requirement is less, but cannot be ignored. I personally wouldn't go
below about 25-50% even with SSD, and I'd plan on 50-100% for regular disks.
There is some evidence to suggest that you only need unallocated RAM
equal to 10% of your index size for caching with SSD, but that is only
likely to work if you have a lot of stored (as opposed to indexed) data.
If most of your index is unstored, then more would be required.
Thanks,
Shawn
RE: Distributing lucene segments across multiple disks.
Posted by Greg Walters <gw...@sherpaanalytics.com>.
Deepak,
It might be a bit outside what you're willing to consider but you can make a raid out of your spinning disks then use your SSD(s) as a dm-cache device to accelerate reads and writes to the raid device. If you're putting lucene indexes on a mixed bag of disks and ssd's without any type of control for what goes where you'd want to use the ssd to accelerate the spinning disks anyway. Check out http://lwn.net/Articles/540996/ for more information on the dm-cache device.
Thanks,
Greg
-----Original Message-----
From: Deepak Konidena [mailto:deepakkoni@gmail.com]
Sent: Wednesday, September 11, 2013 3:57 PM
To: solr-user@lucene.apache.org
Subject: Re: Distributing lucene segments across multiple disks.
I guess at this point in the discussion, I should probably give some more background on why I am doing what I am doing. Having a single Solr shard (multiple segments) on the same disk is posing severe performance problems under load,in that, calls to Solr cause a lot of connection timeouts. When we looked at the ganglia stats for the Solr box, we saw that while memory, cpu and network usage were quite normal, the i/o wait spiked. We are unsure on what caused the i/o wait and why there were no spikes in the cpu/memory usage. Since the Solr box is a beefy box (multi-core setup, huge ram, SSD), we'd like to distribute the segments to multiple locations (disks) and see whether this improves performance under load.
@Greg - Thanks for clarifying that. I just learnt that I can't set them up using RAID as some of them are SSDs and some others are SATA (spinning disks).
@Shawn Heisey - Could you elaborate more about the "broker" core and delegating the requests to other cores?
-Deepak
On Wed, Sep 11, 2013 at 1:10 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 9/11/2013 1:07 PM, Deepak Konidena wrote:
>
>> Are you suggesting a multi-core setup, where all the cores share the
>> same schema, and the cores lie on different disks?
>>
>> Basically, I'd like to know if I can distribute shards/segments on a
>> single machine (with multiple disks) without the use of zookeeper.
>>
>
> Sure, you can do it all manually. At that point you would not be using
> SolrCloud at all, because the way to enable SolrCloud is to tell Solr where
> zookeeper lives.
>
> Without SolrCloud, there is no cluster automation at all. There is no
> "collection" paradigm, you just have cores. You have to send updates to
> the correct core; they not be redirected for you. Similarly, queries will
> not be load balanced automatically. For Java clients, the CloudSolrServer
> object can work seamlessly when servers go down. If you're not using
> SolrCloud, you can't use CloudSolrServer.
>
> You would be in charge of creating the shards parameter yourself. The way
> that I do this on my index is that I have a "broker" core that has no index
> of its own, but its solrconfig.xml has the shards and shards.qt parameters
> in all the request handler definitions. You can also include the parameter
> with the query.
>
> You would also have to handle redundancy yourself, either with replication
> or with independently updated indexes. I use the latter method, because it
> offers a lot more flexibility than replication.
>
> As mentioned in another reply, setting up RAID with a lot of disks may be
> better than trying to split your index up on different filesystems that
> each reside on different disks. I would recommend RAID10 for Solr, and it
> works best if it's hardware RAID and the controller has battery-backed (or
> NVRAM) cache.
>
> Thanks,
> Shawn
>
>
Re: Distributing lucene segments across multiple disks.
Posted by Deepak Konidena <de...@gmail.com>.
I guess at this point in the discussion, I should probably give some more
background on why I am doing what I am doing. Having a single Solr shard
(multiple segments) on the same disk is posing severe performance problems
under load,in that, calls to Solr cause a lot of connection timeouts. When
we looked at the ganglia stats for the Solr box, we saw that while memory,
cpu and network usage were quite normal, the i/o wait spiked. We are unsure
on what caused the i/o wait and why there were no spikes in the cpu/memory
usage. Since the Solr box is a beefy box (multi-core setup, huge ram, SSD),
we'd like to distribute the segments to multiple locations (disks) and see
whether this improves performance under load.
@Greg - Thanks for clarifying that. I just learnt that I can't set them up
using RAID as some of them are SSDs and some others are SATA (spinning
disks).
@Shawn Heisey - Could you elaborate more about the "broker" core and
delegating the requests to other cores?
-Deepak
On Wed, Sep 11, 2013 at 1:10 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 9/11/2013 1:07 PM, Deepak Konidena wrote:
>
>> Are you suggesting a multi-core setup, where all the cores share the same
>> schema, and the cores lie on different disks?
>>
>> Basically, I'd like to know if I can distribute shards/segments on a
>> single
>> machine (with multiple disks) without the use of zookeeper.
>>
>
> Sure, you can do it all manually. At that point you would not be using
> SolrCloud at all, because the way to enable SolrCloud is to tell Solr where
> zookeeper lives.
>
> Without SolrCloud, there is no cluster automation at all. There is no
> "collection" paradigm, you just have cores. You have to send updates to
> the correct core; they not be redirected for you. Similarly, queries will
> not be load balanced automatically. For Java clients, the CloudSolrServer
> object can work seamlessly when servers go down. If you're not using
> SolrCloud, you can't use CloudSolrServer.
>
> You would be in charge of creating the shards parameter yourself. The way
> that I do this on my index is that I have a "broker" core that has no index
> of its own, but its solrconfig.xml has the shards and shards.qt parameters
> in all the request handler definitions. You can also include the parameter
> with the query.
>
> You would also have to handle redundancy yourself, either with replication
> or with independently updated indexes. I use the latter method, because it
> offers a lot more flexibility than replication.
>
> As mentioned in another reply, setting up RAID with a lot of disks may be
> better than trying to split your index up on different filesystems that
> each reside on different disks. I would recommend RAID10 for Solr, and it
> works best if it's hardware RAID and the controller has battery-backed (or
> NVRAM) cache.
>
> Thanks,
> Shawn
>
>
Re: Distributing lucene segments across multiple disks.
Posted by Shawn Heisey <so...@elyograg.org>.
On 9/11/2013 1:07 PM, Deepak Konidena wrote:
> Are you suggesting a multi-core setup, where all the cores share the same
> schema, and the cores lie on different disks?
>
> Basically, I'd like to know if I can distribute shards/segments on a single
> machine (with multiple disks) without the use of zookeeper.
Sure, you can do it all manually. At that point you would not be using
SolrCloud at all, because the way to enable SolrCloud is to tell Solr
where zookeeper lives.
Without SolrCloud, there is no cluster automation at all. There is no
"collection" paradigm, you just have cores. You have to send updates to
the correct core; they not be redirected for you. Similarly, queries
will not be load balanced automatically. For Java clients, the
CloudSolrServer object can work seamlessly when servers go down. If
you're not using SolrCloud, you can't use CloudSolrServer.
You would be in charge of creating the shards parameter yourself. The
way that I do this on my index is that I have a "broker" core that has
no index of its own, but its solrconfig.xml has the shards and shards.qt
parameters in all the request handler definitions. You can also include
the parameter with the query.
You would also have to handle redundancy yourself, either with
replication or with independently updated indexes. I use the latter
method, because it offers a lot more flexibility than replication.
As mentioned in another reply, setting up RAID with a lot of disks may
be better than trying to split your index up on different filesystems
that each reside on different disks. I would recommend RAID10 for Solr,
and it works best if it's hardware RAID and the controller has
battery-backed (or NVRAM) cache.
Thanks,
Shawn
Re: Distributing lucene segments across multiple disks.
Posted by Deepak Konidena <de...@gmail.com>.
Are you suggesting a multi-core setup, where all the cores share the same
schema, and the cores lie on different disks?
Basically, I'd like to know if I can distribute shards/segments on a single
machine (with multiple disks) without the use of zookeeper.
-Deepak
On Wed, Sep 11, 2013 at 11:55 AM, Upayavira <uv...@odoko.co.uk> wrote:
> I think you'll find it hard to distribute different segments between
> disks, as they are typically stored in the same directory.
>
> However, instantiating separate cores on different disks should be
> straight-forward enough, and would give you a performance benefit.
>
> I've certainly heard of that done at Amazon, with a separate EBS volume
> per core giving some performance improvement.
>
> Upayavira
>
> On Wed, Sep 11, 2013, at 07:35 PM, Deepak Konidena wrote:
> > Hi,
> >
> > I know that SolrCloud allows you to have multiple shards on different
> > machines (or a single machine). But it requires a zookeeper installation
> > for doing things like leader election, leader availability, etc
> >
> > While SolrCloud may be the ideal solution for my usecase eventually, I'd
> > like to know if there's a way I can point my Solr instance to read lucene
> > segments distributed across different disks attached to the same machine.
> >
> > Thanks!
> >
> > -Deepak
>
Re: Distributing lucene segments across multiple disks.
Posted by Upayavira <uv...@odoko.co.uk>.
I think you'll find it hard to distribute different segments between
disks, as they are typically stored in the same directory.
However, instantiating separate cores on different disks should be
straight-forward enough, and would give you a performance benefit.
I've certainly heard of that done at Amazon, with a separate EBS volume
per core giving some performance improvement.
Upayavira
On Wed, Sep 11, 2013, at 07:35 PM, Deepak Konidena wrote:
> Hi,
>
> I know that SolrCloud allows you to have multiple shards on different
> machines (or a single machine). But it requires a zookeeper installation
> for doing things like leader election, leader availability, etc
>
> While SolrCloud may be the ideal solution for my usecase eventually, I'd
> like to know if there's a way I can point my Solr instance to read lucene
> segments distributed across different disks attached to the same machine.
>
> Thanks!
>
> -Deepak