You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rich Cariens <ri...@gmail.com> on 2011/08/22 16:54:44 UTC
SSD experience
Ahoy ahoy!
Does anyone have any experiences or stories they can share with the list
about how SSDs impacted search performance for better or worse?
I found a Lucene SSD performance benchmark
doc<http://wiki.apache.org/lucene-java/SSD_performance?action=AttachFile&do=view&target=combined-disk-ssd.pdf>but
the wiki engine is refusing to let me view the attachment (I get "You
are not allowed to do AttachFile on this page.").
Thanks in advance!
Re: SSD experience
Posted by Rich Cariens <ri...@gmail.com>.
Thanks folks!
On Mon, Aug 22, 2011 at 11:13 AM, Erick Erickson <er...@gmail.com>wrote:
> That link appears to be foo'd, and I can't find the original doc.
>
> But others (mostly on the user's list historically) have seen very
> significant
> performance improvements with SSDs, *IF* the entire index doesn't fit
> in memory.
>
> If your index does fit entirely in memory, there will probably be some
> improvement when fetching stored fields, especially if the stored fields
> are large. But I'm not sure the cost is worth the incremental speed
> in this case.. Of course if you can get your IT folks to spring for SSDs,
> go for it :)
>
> Best
> Erick
>
> On Mon, Aug 22, 2011 at 11:02 AM, Daniel Skiles
> <da...@docfinity.com> wrote:
> > I haven't tried it with Solr yet, but with straight Lucene about two
> years
> > ago we saw about a 40% boost in performance on our tests with no changes
> > except the disk.
> >
> > On Mon, Aug 22, 2011 at 10:54 AM, Rich Cariens <richcariens@gmail.com
> >wrote:
> >
> >> Ahoy ahoy!
> >>
> >> Does anyone have any experiences or stories they can share with the list
> >> about how SSDs impacted search performance for better or worse?
> >>
> >> I found a Lucene SSD performance benchmark
> >> doc<
> >>
> http://wiki.apache.org/lucene-java/SSD_performance?action=AttachFile&do=view&target=combined-disk-ssd.pdf
> >> >but
> >> the wiki engine is refusing to let me view the attachment (I get "You
> >> are not allowed to do AttachFile on this page.").
> >>
> >> Thanks in advance!
> >>
> >
>
Re: SSD experience
Posted by Erick Erickson <er...@gmail.com>.
That link appears to be foo'd, and I can't find the original doc.
But others (mostly on the user's list historically) have seen very significant
performance improvements with SSDs, *IF* the entire index doesn't fit
in memory.
If your index does fit entirely in memory, there will probably be some
improvement when fetching stored fields, especially if the stored fields
are large. But I'm not sure the cost is worth the incremental speed
in this case.. Of course if you can get your IT folks to spring for SSDs,
go for it :)
Best
Erick
On Mon, Aug 22, 2011 at 11:02 AM, Daniel Skiles
<da...@docfinity.com> wrote:
> I haven't tried it with Solr yet, but with straight Lucene about two years
> ago we saw about a 40% boost in performance on our tests with no changes
> except the disk.
>
> On Mon, Aug 22, 2011 at 10:54 AM, Rich Cariens <ri...@gmail.com>wrote:
>
>> Ahoy ahoy!
>>
>> Does anyone have any experiences or stories they can share with the list
>> about how SSDs impacted search performance for better or worse?
>>
>> I found a Lucene SSD performance benchmark
>> doc<
>> http://wiki.apache.org/lucene-java/SSD_performance?action=AttachFile&do=view&target=combined-disk-ssd.pdf
>> >but
>> the wiki engine is refusing to let me view the attachment (I get "You
>> are not allowed to do AttachFile on this page.").
>>
>> Thanks in advance!
>>
>
Re: SSD experience
Posted by Daniel Skiles <da...@docfinity.com>.
I haven't tried it with Solr yet, but with straight Lucene about two years
ago we saw about a 40% boost in performance on our tests with no changes
except the disk.
On Mon, Aug 22, 2011 at 10:54 AM, Rich Cariens <ri...@gmail.com>wrote:
> Ahoy ahoy!
>
> Does anyone have any experiences or stories they can share with the list
> about how SSDs impacted search performance for better or worse?
>
> I found a Lucene SSD performance benchmark
> doc<
> http://wiki.apache.org/lucene-java/SSD_performance?action=AttachFile&do=view&target=combined-disk-ssd.pdf
> >but
> the wiki engine is refusing to let me view the attachment (I get "You
> are not allowed to do AttachFile on this page.").
>
> Thanks in advance!
>
Re: SSD experience
Posted by Peter Sturge <pe...@gmail.com>.
Ah yes, the beautiful new links in Windows 6. These are 'symlinks' in
name only - they operate *very* differently from LUNIX symlinks, and
sadly, not quite so well. NTFS is one of the best things about
Windows, but it's architecture is not well suited to 'on-the-fly'
redirection, as there are many items 'in the chain' to cater for at
various points - e.g. driver stack, sid context, SACL/DACLs, DFS,
auditing etc.This makes links on NTFS much more difficult to manage
and it is common to encounter all manner of strange behaviour when
using them.
On Tue, Aug 23, 2011 at 5:34 PM, Sanne Grinovero
<sa...@gmail.com> wrote:
> Indeed I would never actually use it, but symlinks do exist on Windows.
>
> http://en.wikipedia.org/wiki/NTFS_symbolic_link
>
> Sanne
>
> 2011/8/23 Peter Sturge <pe...@gmail.com>:
>> The Solr index directory lives directly on the SSD (running on Windows
>> - where the word symlink does not appear in any dictionary within a
>> 100 mile radius of Redmond :-)
>>
>> Currently, the main limiting factors of SSD are cost and size. SSDs
>> will get larger over time. Splitting indexes across multiple shards on
>> multiple SSDs is a wonderfully fast, if not slightly extravagant
>> method of getting excellent IO performance.
>> Regarding cost, I've seen many organizations where the use of fast
>> SANs costs at least the same if not more per GB of storage than SSD.
>> Hybrid drives can be a good cost-effective alternative as well.
>>
>> Peter
>>
>>
>>
>> On Tue, Aug 23, 2011 at 3:29 PM, Gerard Roos <li...@gerardroos.nl> wrote:
>>> Interesting. Do you make a symlink to the indexes or is the whole Solr directory on SSD?
>>>
>>> thanks,
>>> Gerard
>>>
>>> Op 23 aug. 2011, om 12:53 heeft Peter Sturge het volgende geschreven:
>>>
>>>> Just to add a few cents worth regarding SSD...
>>>>
>>>> We use Vertex SSD drives for storing indexes, and wow, they really
>>>> scream compared to SATA/SAS/SAN. As we do some heavy commits, it's the
>>>> commit times where we see the biggest performance boost.
>>>> In tests, we found that locally attached 15k SAS drives are the next
>>>> best for performance. SANs can work well, but should be FibreChannel.
>>>> IP-based SANs are ok, as long they're not heavily taxed by other,
>>>> non-Solr disk I/O.
>>>> NAS is far and away the poorest performing - not recommended for real indexes.
>>>>
>>>> HTH,
>>>> Peter
>>>>
>>>>
>>>>
>>>> On Mon, Aug 22, 2011 at 3:54 PM, Rich Cariens <ri...@gmail.com> wrote:
>>>>> Ahoy ahoy!
>>>>>
>>>>> Does anyone have any experiences or stories they can share with the list
>>>>> about how SSDs impacted search performance for better or worse?
>>>>>
>>>>> I found a Lucene SSD performance benchmark
>>>>> doc<http://wiki.apache.org/lucene-java/SSD_performance?action=AttachFile&do=view&target=combined-disk-ssd.pdf>but
>>>>> the wiki engine is refusing to let me view the attachment (I get "You
>>>>> are not allowed to do AttachFile on this page.").
>>>>>
>>>>> Thanks in advance!
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
Re: SSD experience
Posted by Sanne Grinovero <sa...@gmail.com>.
Indeed I would never actually use it, but symlinks do exist on Windows.
http://en.wikipedia.org/wiki/NTFS_symbolic_link
Sanne
2011/8/23 Peter Sturge <pe...@gmail.com>:
> The Solr index directory lives directly on the SSD (running on Windows
> - where the word symlink does not appear in any dictionary within a
> 100 mile radius of Redmond :-)
>
> Currently, the main limiting factors of SSD are cost and size. SSDs
> will get larger over time. Splitting indexes across multiple shards on
> multiple SSDs is a wonderfully fast, if not slightly extravagant
> method of getting excellent IO performance.
> Regarding cost, I've seen many organizations where the use of fast
> SANs costs at least the same if not more per GB of storage than SSD.
> Hybrid drives can be a good cost-effective alternative as well.
>
> Peter
>
>
>
> On Tue, Aug 23, 2011 at 3:29 PM, Gerard Roos <li...@gerardroos.nl> wrote:
>> Interesting. Do you make a symlink to the indexes or is the whole Solr directory on SSD?
>>
>> thanks,
>> Gerard
>>
>> Op 23 aug. 2011, om 12:53 heeft Peter Sturge het volgende geschreven:
>>
>>> Just to add a few cents worth regarding SSD...
>>>
>>> We use Vertex SSD drives for storing indexes, and wow, they really
>>> scream compared to SATA/SAS/SAN. As we do some heavy commits, it's the
>>> commit times where we see the biggest performance boost.
>>> In tests, we found that locally attached 15k SAS drives are the next
>>> best for performance. SANs can work well, but should be FibreChannel.
>>> IP-based SANs are ok, as long they're not heavily taxed by other,
>>> non-Solr disk I/O.
>>> NAS is far and away the poorest performing - not recommended for real indexes.
>>>
>>> HTH,
>>> Peter
>>>
>>>
>>>
>>> On Mon, Aug 22, 2011 at 3:54 PM, Rich Cariens <ri...@gmail.com> wrote:
>>>> Ahoy ahoy!
>>>>
>>>> Does anyone have any experiences or stories they can share with the list
>>>> about how SSDs impacted search performance for better or worse?
>>>>
>>>> I found a Lucene SSD performance benchmark
>>>> doc<http://wiki.apache.org/lucene-java/SSD_performance?action=AttachFile&do=view&target=combined-disk-ssd.pdf>but
>>>> the wiki engine is refusing to let me view the attachment (I get "You
>>>> are not allowed to do AttachFile on this page.").
>>>>
>>>> Thanks in advance!
>>>>
>>>
>>>
>>
>>
>
Re: SSD experience
Posted by Peter Sturge <pe...@gmail.com>.
The Solr index directory lives directly on the SSD (running on Windows
- where the word symlink does not appear in any dictionary within a
100 mile radius of Redmond :-)
Currently, the main limiting factors of SSD are cost and size. SSDs
will get larger over time. Splitting indexes across multiple shards on
multiple SSDs is a wonderfully fast, if not slightly extravagant
method of getting excellent IO performance.
Regarding cost, I've seen many organizations where the use of fast
SANs costs at least the same if not more per GB of storage than SSD.
Hybrid drives can be a good cost-effective alternative as well.
Peter
On Tue, Aug 23, 2011 at 3:29 PM, Gerard Roos <li...@gerardroos.nl> wrote:
> Interesting. Do you make a symlink to the indexes or is the whole Solr directory on SSD?
>
> thanks,
> Gerard
>
> Op 23 aug. 2011, om 12:53 heeft Peter Sturge het volgende geschreven:
>
>> Just to add a few cents worth regarding SSD...
>>
>> We use Vertex SSD drives for storing indexes, and wow, they really
>> scream compared to SATA/SAS/SAN. As we do some heavy commits, it's the
>> commit times where we see the biggest performance boost.
>> In tests, we found that locally attached 15k SAS drives are the next
>> best for performance. SANs can work well, but should be FibreChannel.
>> IP-based SANs are ok, as long they're not heavily taxed by other,
>> non-Solr disk I/O.
>> NAS is far and away the poorest performing - not recommended for real indexes.
>>
>> HTH,
>> Peter
>>
>>
>>
>> On Mon, Aug 22, 2011 at 3:54 PM, Rich Cariens <ri...@gmail.com> wrote:
>>> Ahoy ahoy!
>>>
>>> Does anyone have any experiences or stories they can share with the list
>>> about how SSDs impacted search performance for better or worse?
>>>
>>> I found a Lucene SSD performance benchmark
>>> doc<http://wiki.apache.org/lucene-java/SSD_performance?action=AttachFile&do=view&target=combined-disk-ssd.pdf>but
>>> the wiki engine is refusing to let me view the attachment (I get "You
>>> are not allowed to do AttachFile on this page.").
>>>
>>> Thanks in advance!
>>>
>>
>>
>
>
Re: SSD experience
Posted by Gerard Roos <li...@gerardroos.nl>.
Interesting. Do you make a symlink to the indexes or is the whole Solr directory on SSD?
thanks,
Gerard
Op 23 aug. 2011, om 12:53 heeft Peter Sturge het volgende geschreven:
> Just to add a few cents worth regarding SSD...
>
> We use Vertex SSD drives for storing indexes, and wow, they really
> scream compared to SATA/SAS/SAN. As we do some heavy commits, it's the
> commit times where we see the biggest performance boost.
> In tests, we found that locally attached 15k SAS drives are the next
> best for performance. SANs can work well, but should be FibreChannel.
> IP-based SANs are ok, as long they're not heavily taxed by other,
> non-Solr disk I/O.
> NAS is far and away the poorest performing - not recommended for real indexes.
>
> HTH,
> Peter
>
>
>
> On Mon, Aug 22, 2011 at 3:54 PM, Rich Cariens <ri...@gmail.com> wrote:
>> Ahoy ahoy!
>>
>> Does anyone have any experiences or stories they can share with the list
>> about how SSDs impacted search performance for better or worse?
>>
>> I found a Lucene SSD performance benchmark
>> doc<http://wiki.apache.org/lucene-java/SSD_performance?action=AttachFile&do=view&target=combined-disk-ssd.pdf>but
>> the wiki engine is refusing to let me view the attachment (I get "You
>> are not allowed to do AttachFile on this page.").
>>
>> Thanks in advance!
>>
>
>
Re: SSD experience
Posted by Peter Sturge <pe...@gmail.com>.
Just to add a few cents worth regarding SSD...
We use Vertex SSD drives for storing indexes, and wow, they really
scream compared to SATA/SAS/SAN. As we do some heavy commits, it's the
commit times where we see the biggest performance boost.
In tests, we found that locally attached 15k SAS drives are the next
best for performance. SANs can work well, but should be FibreChannel.
IP-based SANs are ok, as long they're not heavily taxed by other,
non-Solr disk I/O.
NAS is far and away the poorest performing - not recommended for real indexes.
HTH,
Peter
On Mon, Aug 22, 2011 at 3:54 PM, Rich Cariens <ri...@gmail.com> wrote:
> Ahoy ahoy!
>
> Does anyone have any experiences or stories they can share with the list
> about how SSDs impacted search performance for better or worse?
>
> I found a Lucene SSD performance benchmark
> doc<http://wiki.apache.org/lucene-java/SSD_performance?action=AttachFile&do=view&target=combined-disk-ssd.pdf>but
> the wiki engine is refusing to let me view the attachment (I get "You
> are not allowed to do AttachFile on this page.").
>
> Thanks in advance!
>