You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Uwe Schindler <uw...@thetaphi.de> on 2013/08/19 17:03:53 UTC
Lucene tests killed one other SSD - Policeman Jenkins
Hi,
there were some problems with Policeman Jenkins the last days. The server died 6 times the last month, recently 2 times in 24 hours. After I moved away the swap file from the SSD, the failures were no longer fatal for the server but fatal for some Jenkins runs :-)
Finally the SSD device got unresponsible and only after a power cycle it was responsible again. The error messages in dmesg look similar to other dying OCX Vertex 2 drives.
Now the statistics: During the whole lifetime of this SSD (2.5 years; which is the lifetime of the server), it was mostly unused (it was just a "addon", provided by the hosting provider, thanks to Serverloft / Plusserver). 1.5 years ago, Robert Muir and also Mike McCandless decided to use the server of my own company SD DataSolutions to do more than idling most of the time: We installed Jenkins and 2 additional virtualbox machines on this server after the 2012 Lucene Revolution conference and the "spare" SSD was given as base for swap file, Jenkins Workspace and virtual disks for the Windows and Haskintosh machines.
During this time (1 year, 3 months) the SSD did hard work, according to SMART:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 112 112 050 Pre-fail Always - 0/61244435
5 Retired_Block_Count 0x0033 100 100 003 Pre-fail Always - 0
9 Power_On_Hours_and_Msec 0x0032 100 100 000 Old_age Always - 21904h+48m+22.180s
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 19
171 Program_Fail_Count 0x0032 000 000 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 000 000 000 Old_age Always - 0
174 Unexpect_Power_Loss_Ct 0x0030 000 000 000 Old_age Offline - 6
177 Wear_Range_Delta 0x0000 000 000 000 Old_age Offline - 2
181 Program_Fail_Count 0x0032 000 000 000 Old_age Always - 0
182 Erase_Fail_Count 0x0032 000 000 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
194 Temperature_Celsius 0x0022 001 129 000 Old_age Always - 1 (0 127 0 129)
195 ECC_Uncorr_Error_Count 0x001c 112 112 000 Old_age Offline - 0/61244435
196 Reallocated_Event_Count 0x0033 100 100 000 Pre-fail Always - 0
231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 0
233 SandForce_Internal 0x0000 000 000 000 Old_age Offline - 18752
234 SandForce_Internal 0x0032 000 000 000 Old_age Always - 53376
241 Lifetime_Writes_GiB 0x0032 000 000 000 Old_age Always - 53376
242 Lifetime_Reads_GiB 0x0032 000 000 000 Old_age Always - 22784
Last 2 lines are interesting:
53 Terabytes written to it and 22 Terabytes read from it. Ignore swap (mostly unused as swappiness is low), so our tests are reading and writing a lot!
And unfortunately after that it died (or almost died) this morning. Cause is unclear, it could also be broken SATA cable, but from the web the given error messages in "dmesg" seem to also be caused by drive failure (especially as it is a timeout, not DMA error)! See https://paste.apache.org/bjAH
So just to conclude: Lucene kills SSDs :-) Mike still has one Vertex 3 running (his Intel one died before).
Of course as this is a rented server, the hosting provider will replace the SSD (I was able to copy the data off, but the Jenkins workspace is not really important data, more the virtual machines). After that one more year with a new SSD, or should it survive longer? Let's see what type I will get as replacement. I have no idea when it is replaced, so excuse any jenkins downtime and after that maybe broken builds until all is settled again. At the moment Jenkins is running much slower from the RAID 1 harddisks (with lots of IOWAITS!).
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene tests killed one other SSD - Policeman Jenkins
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Mon, 2013-08-19 at 17:03 +0200, Uwe Schindler wrote:
> Finally the SSD device got unresponsible and only after a power cycle
> it was responsible again. The error messages in dmesg look similar to
> other dying OCX Vertex 2 drives.
The only statistics I could find was
http://www.tomshardware.com/reviews/ssd-reliability-failure-rate,2923-3.html
It is a bit old and does not speak well for the Vertex 2 series.
> So just to conclude: Lucene kills SSDs :-)
I am an accomplice to murder!? Oh Noes!
- Toke Eskildsen, happily using an old 160GB Intel X25 SSD with 11TB
written and 3 reallocated sectors.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: Lucene tests killed one other SSD - Policeman Jenkins
Posted by Uwe Schindler <uw...@thetaphi.de>.
By the way, the value of 096 did not change the last months, so the recent failures seem to be something elase.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, August 19, 2013 6:31 PM
> To: 'dev@lucene.apache.org'
> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
>
> Hi Karl,
>
> Only 70 full writes seems a little bit low for an SSD. This SSD has a capacity of
> 64 Gigabytes so 53 Terabytes written is not bad for it. It already had the first
> few errors, so you might be right, as you see from this Smart data:
>
> 231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 0
>
> So this "arbitrary" counter just went down from 100 to 096. I have the feeling
> that the current issues with this SSD are not really write problems, more
> hardware interface or firmware problems (as seen quite often for OCZ
> Vertex 2). Otherwise it would fail very early. According this data it should
> survive more 10th of years?
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: karl.wright@here.com [mailto:karl.wright@here.com]
> > Sent: Monday, August 19, 2013 5:07 PM
> > To: dev@lucene.apache.org
> > Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
> >
> > I am told that SSD's are spec'd for only 70 full writes before they
> > get an error. The error block is set aside but eventually something
> > critical gets hit. So you should probably should expect this to happen again.
> >
> > Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene tests killed one other SSD - Policeman Jenkins
Posted by Michael McCandless <lu...@mikemccandless.com>.
I see, so the 70X is how soon you can expect to see your first
reallocated sector?
But, that still seems way too low; the underlying erase/write cycles
on the NAND cells is in the 1000s I think? Hmm looks like for modern
20nm NAND cells it's ~1000.
If the wear leveling is working why should you see reallocated sectors
after only 70X writes? My OCZ Vertex 3 is at 341X and no reallocated
sectors so far!
Mike McCandless
http://blog.mikemccandless.com
On Mon, Aug 19, 2013 at 1:09 PM, <ka...@here.com> wrote:
> Right, that's what I said. And one write means writing the *whole* disk. So Mike and I may *both* be right. ;-)
>
> Karl
>
> -----Original Message-----
> From: ext Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, August 19, 2013 1:07 PM
> To: dev@lucene.apache.org
> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
>
> Hi,
>
> I misunderstood this number on the first time, too.
>
> 70x does not mean 70 times write fully and then the SSD is gone. It means 70 times write of the SSD capacity until the first error somewhere in the flash memory occurs. Those errors are not fatal, they are just list in the "relocated sector count" or decrease this "SSD remaining lifetime" smart attribute. So 70 might be correct.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Michael McCandless [mailto:lucene@mikemccandless.com]
>> Sent: Monday, August 19, 2013 6:57 PM
>> To: Lucene/Solr dev
>> Subject: Re: Lucene tests killed one other SSD - Policeman Jenkins
>>
>> I am curious :) 70X is really way too low.
>>
>> I just checked the OCZ Vertex 3 behind the nightly Lucene performance
>> tests ( http://people.apache.org/~mikemccand/lucenebench/index.html ),
>> which indexes the full Wikpedia English export 3 times nightly.
>>
>> It's written 74.4 TB, read 35.7 TB, and its odometer (SSD_Life_Left
>> from
>> smartctl) says 98 out of 100 ... which I think equates to ~11,000x -
>> 17,000x write endurance.
>>
>> This thread is also fun:
>>
>> http://www.xtremesystems.org/forums/showthread.php?271063-SSD-
>> Write-Endurance-25nm-Vs-34nm
>>
>> Users destroy their SSDs by writing and writing and writing until it
>> dies :)
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Mon, Aug 19, 2013 at 12:50 PM, <ka...@here.com> wrote:
>> > Mike, I'm talking about a 1TB SSD option for some hardware we are buying.
>> If you are really curious, I can ask the people who are doing the
>> project for the model and specs.
>> >
>> > Karl
>> >
>> > -----Original Message-----
>> > From: ext Michael McCandless [mailto:lucene@mikemccandless.com]
>> > Sent: Monday, August 19, 2013 12:47 PM
>> > To: Lucene/Solr dev
>> > Subject: Re: Lucene tests killed one other SSD - Policeman Jenkins
>> >
>> > Er, which drives have write endurance only 70x? That's exceptionally low?
>> I thought this was more like ~1000x for modern consumer SSDs?
>> >
>> > Mike McCandless
>> >
>> > http://blog.mikemccandless.com
>> >
>> >
>> > On Mon, Aug 19, 2013 at 12:35 PM, <ka...@here.com> wrote:
>> >> " Only 70 full writes seems a little bit low for an SSD."
>> >>
>> >> That's what I thought. I was astounded to learn that that is in
>> >> fact correct
>> (at least for some of the drives we are using here). Automatic
>> recovery is how the SSD copes with this failure rate.
>> >>
>> >> But it is entirely possible that the cause of your problems are
>> >> something
>> else.
>> >>
>> >> Karl
>> >>
>> >> -----Original Message-----
>> >> From: ext Uwe Schindler [mailto:uwe@thetaphi.de]
>> >> Sent: Monday, August 19, 2013 12:31 PM
>> >> To: dev@lucene.apache.org
>> >> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
>> >>
>> >> Hi Karl,
>> >>
>> >> Only 70 full writes seems a little bit low for an SSD. This SSD has
>> >> a capacity
>> of 64 Gigabytes so 53 Terabytes written is not bad for it. It already
>> had the first few errors, so you might be right, as you see from this Smart data:
>> >>
>> >> 231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 0
>> >>
>> >> So this "arbitrary" counter just went down from 100 to 096. I have
>> >> the
>> feeling that the current issues with this SSD are not really write
>> problems, more hardware interface or firmware problems (as seen quite
>> often for OCZ Vertex 2). Otherwise it would fail very early. According
>> this data it should survive more 10th of years?
>> >>
>> >> Uwe
>> >>
>> >> -----
>> >> Uwe Schindler
>> >> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
>> >> eMail: uwe@thetaphi.de
>> >>
>> >>
>> >>> -----Original Message-----
>> >>> From: karl.wright@here.com [mailto:karl.wright@here.com]
>> >>> Sent: Monday, August 19, 2013 5:07 PM
>> >>> To: dev@lucene.apache.org
>> >>> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
>> >>>
>> >>> I am told that SSD's are spec'd for only 70 full writes before they
>> >>> get an error. The error block is set aside but eventually something
>> >>> critical gets hit. So you should probably should expect this to happen
>> again.
>> >>>
>> >>> Karl
>> >>
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
>> >> additional commands, e-mail: dev-help@lucene.apache.org
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
>> > additional commands, e-mail: dev-help@lucene.apache.org
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
>> > additional commands, e-mail: dev-help@lucene.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
>> commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: Lucene tests killed one other SSD - Policeman Jenkins
Posted by ka...@here.com.
Right, that's what I said. And one write means writing the *whole* disk. So Mike and I may *both* be right. ;-)
Karl
-----Original Message-----
From: ext Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Monday, August 19, 2013 1:07 PM
To: dev@lucene.apache.org
Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
Hi,
I misunderstood this number on the first time, too.
70x does not mean 70 times write fully and then the SSD is gone. It means 70 times write of the SSD capacity until the first error somewhere in the flash memory occurs. Those errors are not fatal, they are just list in the "relocated sector count" or decrease this "SSD remaining lifetime" smart attribute. So 70 might be correct.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Monday, August 19, 2013 6:57 PM
> To: Lucene/Solr dev
> Subject: Re: Lucene tests killed one other SSD - Policeman Jenkins
>
> I am curious :) 70X is really way too low.
>
> I just checked the OCZ Vertex 3 behind the nightly Lucene performance
> tests ( http://people.apache.org/~mikemccand/lucenebench/index.html ),
> which indexes the full Wikpedia English export 3 times nightly.
>
> It's written 74.4 TB, read 35.7 TB, and its odometer (SSD_Life_Left
> from
> smartctl) says 98 out of 100 ... which I think equates to ~11,000x -
> 17,000x write endurance.
>
> This thread is also fun:
>
> http://www.xtremesystems.org/forums/showthread.php?271063-SSD-
> Write-Endurance-25nm-Vs-34nm
>
> Users destroy their SSDs by writing and writing and writing until it
> dies :)
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Mon, Aug 19, 2013 at 12:50 PM, <ka...@here.com> wrote:
> > Mike, I'm talking about a 1TB SSD option for some hardware we are buying.
> If you are really curious, I can ask the people who are doing the
> project for the model and specs.
> >
> > Karl
> >
> > -----Original Message-----
> > From: ext Michael McCandless [mailto:lucene@mikemccandless.com]
> > Sent: Monday, August 19, 2013 12:47 PM
> > To: Lucene/Solr dev
> > Subject: Re: Lucene tests killed one other SSD - Policeman Jenkins
> >
> > Er, which drives have write endurance only 70x? That's exceptionally low?
> I thought this was more like ~1000x for modern consumer SSDs?
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> >
> > On Mon, Aug 19, 2013 at 12:35 PM, <ka...@here.com> wrote:
> >> " Only 70 full writes seems a little bit low for an SSD."
> >>
> >> That's what I thought. I was astounded to learn that that is in
> >> fact correct
> (at least for some of the drives we are using here). Automatic
> recovery is how the SSD copes with this failure rate.
> >>
> >> But it is entirely possible that the cause of your problems are
> >> something
> else.
> >>
> >> Karl
> >>
> >> -----Original Message-----
> >> From: ext Uwe Schindler [mailto:uwe@thetaphi.de]
> >> Sent: Monday, August 19, 2013 12:31 PM
> >> To: dev@lucene.apache.org
> >> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
> >>
> >> Hi Karl,
> >>
> >> Only 70 full writes seems a little bit low for an SSD. This SSD has
> >> a capacity
> of 64 Gigabytes so 53 Terabytes written is not bad for it. It already
> had the first few errors, so you might be right, as you see from this Smart data:
> >>
> >> 231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 0
> >>
> >> So this "arbitrary" counter just went down from 100 to 096. I have
> >> the
> feeling that the current issues with this SSD are not really write
> problems, more hardware interface or firmware problems (as seen quite
> often for OCZ Vertex 2). Otherwise it would fail very early. According
> this data it should survive more 10th of years?
> >>
> >> Uwe
> >>
> >> -----
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> >> eMail: uwe@thetaphi.de
> >>
> >>
> >>> -----Original Message-----
> >>> From: karl.wright@here.com [mailto:karl.wright@here.com]
> >>> Sent: Monday, August 19, 2013 5:07 PM
> >>> To: dev@lucene.apache.org
> >>> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
> >>>
> >>> I am told that SSD's are spec'd for only 70 full writes before they
> >>> get an error. The error block is set aside but eventually something
> >>> critical gets hit. So you should probably should expect this to happen
> again.
> >>>
> >>> Karl
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> >> additional commands, e-mail: dev-help@lucene.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> > additional commands, e-mail: dev-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> > additional commands, e-mail: dev-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: Lucene tests killed one other SSD - Policeman Jenkins
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
I misunderstood this number on the first time, too.
70x does not mean 70 times write fully and then the SSD is gone. It means 70 times write of the SSD capacity until the first error somewhere in the flash memory occurs. Those errors are not fatal, they are just list in the "relocated sector count" or decrease this "SSD remaining lifetime" smart attribute. So 70 might be correct.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Monday, August 19, 2013 6:57 PM
> To: Lucene/Solr dev
> Subject: Re: Lucene tests killed one other SSD - Policeman Jenkins
>
> I am curious :) 70X is really way too low.
>
> I just checked the OCZ Vertex 3 behind the nightly Lucene performance tests
> ( http://people.apache.org/~mikemccand/lucenebench/index.html ), which
> indexes the full Wikpedia English export 3 times nightly.
>
> It's written 74.4 TB, read 35.7 TB, and its odometer (SSD_Life_Left from
> smartctl) says 98 out of 100 ... which I think equates to ~11,000x - 17,000x
> write endurance.
>
> This thread is also fun:
>
> http://www.xtremesystems.org/forums/showthread.php?271063-SSD-
> Write-Endurance-25nm-Vs-34nm
>
> Users destroy their SSDs by writing and writing and writing until it dies :)
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Mon, Aug 19, 2013 at 12:50 PM, <ka...@here.com> wrote:
> > Mike, I'm talking about a 1TB SSD option for some hardware we are buying.
> If you are really curious, I can ask the people who are doing the project for
> the model and specs.
> >
> > Karl
> >
> > -----Original Message-----
> > From: ext Michael McCandless [mailto:lucene@mikemccandless.com]
> > Sent: Monday, August 19, 2013 12:47 PM
> > To: Lucene/Solr dev
> > Subject: Re: Lucene tests killed one other SSD - Policeman Jenkins
> >
> > Er, which drives have write endurance only 70x? That's exceptionally low?
> I thought this was more like ~1000x for modern consumer SSDs?
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> >
> > On Mon, Aug 19, 2013 at 12:35 PM, <ka...@here.com> wrote:
> >> " Only 70 full writes seems a little bit low for an SSD."
> >>
> >> That's what I thought. I was astounded to learn that that is in fact correct
> (at least for some of the drives we are using here). Automatic recovery is
> how the SSD copes with this failure rate.
> >>
> >> But it is entirely possible that the cause of your problems are something
> else.
> >>
> >> Karl
> >>
> >> -----Original Message-----
> >> From: ext Uwe Schindler [mailto:uwe@thetaphi.de]
> >> Sent: Monday, August 19, 2013 12:31 PM
> >> To: dev@lucene.apache.org
> >> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
> >>
> >> Hi Karl,
> >>
> >> Only 70 full writes seems a little bit low for an SSD. This SSD has a capacity
> of 64 Gigabytes so 53 Terabytes written is not bad for it. It already had the
> first few errors, so you might be right, as you see from this Smart data:
> >>
> >> 231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 0
> >>
> >> So this "arbitrary" counter just went down from 100 to 096. I have the
> feeling that the current issues with this SSD are not really write problems,
> more hardware interface or firmware problems (as seen quite often for OCZ
> Vertex 2). Otherwise it would fail very early. According this data it should
> survive more 10th of years?
> >>
> >> Uwe
> >>
> >> -----
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: uwe@thetaphi.de
> >>
> >>
> >>> -----Original Message-----
> >>> From: karl.wright@here.com [mailto:karl.wright@here.com]
> >>> Sent: Monday, August 19, 2013 5:07 PM
> >>> To: dev@lucene.apache.org
> >>> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
> >>>
> >>> I am told that SSD's are spec'd for only 70 full writes before they
> >>> get an error. The error block is set aside but eventually something
> >>> critical gets hit. So you should probably should expect this to happen
> again.
> >>>
> >>> Karl
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> >> additional commands, e-mail: dev-help@lucene.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> > additional commands, e-mail: dev-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> > additional commands, e-mail: dev-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene tests killed one other SSD - Policeman Jenkins
Posted by Michael McCandless <lu...@mikemccandless.com>.
I am curious :) 70X is really way too low.
I just checked the OCZ Vertex 3 behind the nightly Lucene performance
tests ( http://people.apache.org/~mikemccand/lucenebench/index.html ),
which indexes the full Wikpedia English export 3 times nightly.
It's written 74.4 TB, read 35.7 TB, and its odometer (SSD_Life_Left
from smartctl) says 98 out of 100 ... which I think equates to
~11,000x - 17,000x write endurance.
This thread is also fun:
http://www.xtremesystems.org/forums/showthread.php?271063-SSD-Write-Endurance-25nm-Vs-34nm
Users destroy their SSDs by writing and writing and writing until it dies :)
Mike McCandless
http://blog.mikemccandless.com
On Mon, Aug 19, 2013 at 12:50 PM, <ka...@here.com> wrote:
> Mike, I'm talking about a 1TB SSD option for some hardware we are buying. If you are really curious, I can ask the people who are doing the project for the model and specs.
>
> Karl
>
> -----Original Message-----
> From: ext Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Monday, August 19, 2013 12:47 PM
> To: Lucene/Solr dev
> Subject: Re: Lucene tests killed one other SSD - Policeman Jenkins
>
> Er, which drives have write endurance only 70x? That's exceptionally low? I thought this was more like ~1000x for modern consumer SSDs?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Mon, Aug 19, 2013 at 12:35 PM, <ka...@here.com> wrote:
>> " Only 70 full writes seems a little bit low for an SSD."
>>
>> That's what I thought. I was astounded to learn that that is in fact correct (at least for some of the drives we are using here). Automatic recovery is how the SSD copes with this failure rate.
>>
>> But it is entirely possible that the cause of your problems are something else.
>>
>> Karl
>>
>> -----Original Message-----
>> From: ext Uwe Schindler [mailto:uwe@thetaphi.de]
>> Sent: Monday, August 19, 2013 12:31 PM
>> To: dev@lucene.apache.org
>> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
>>
>> Hi Karl,
>>
>> Only 70 full writes seems a little bit low for an SSD. This SSD has a capacity of 64 Gigabytes so 53 Terabytes written is not bad for it. It already had the first few errors, so you might be right, as you see from this Smart data:
>>
>> 231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 0
>>
>> So this "arbitrary" counter just went down from 100 to 096. I have the feeling that the current issues with this SSD are not really write problems, more hardware interface or firmware problems (as seen quite often for OCZ Vertex 2). Otherwise it would fail very early. According this data it should survive more 10th of years?
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>>> -----Original Message-----
>>> From: karl.wright@here.com [mailto:karl.wright@here.com]
>>> Sent: Monday, August 19, 2013 5:07 PM
>>> To: dev@lucene.apache.org
>>> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
>>>
>>> I am told that SSD's are spec'd for only 70 full writes before they
>>> get an error. The error block is set aside but eventually something
>>> critical gets hit. So you should probably should expect this to happen again.
>>>
>>> Karl
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
>> additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: Lucene tests killed one other SSD - Policeman Jenkins
Posted by ka...@here.com.
Mike, I'm talking about a 1TB SSD option for some hardware we are buying. If you are really curious, I can ask the people who are doing the project for the model and specs.
Karl
-----Original Message-----
From: ext Michael McCandless [mailto:lucene@mikemccandless.com]
Sent: Monday, August 19, 2013 12:47 PM
To: Lucene/Solr dev
Subject: Re: Lucene tests killed one other SSD - Policeman Jenkins
Er, which drives have write endurance only 70x? That's exceptionally low? I thought this was more like ~1000x for modern consumer SSDs?
Mike McCandless
http://blog.mikemccandless.com
On Mon, Aug 19, 2013 at 12:35 PM, <ka...@here.com> wrote:
> " Only 70 full writes seems a little bit low for an SSD."
>
> That's what I thought. I was astounded to learn that that is in fact correct (at least for some of the drives we are using here). Automatic recovery is how the SSD copes with this failure rate.
>
> But it is entirely possible that the cause of your problems are something else.
>
> Karl
>
> -----Original Message-----
> From: ext Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, August 19, 2013 12:31 PM
> To: dev@lucene.apache.org
> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
>
> Hi Karl,
>
> Only 70 full writes seems a little bit low for an SSD. This SSD has a capacity of 64 Gigabytes so 53 Terabytes written is not bad for it. It already had the first few errors, so you might be right, as you see from this Smart data:
>
> 231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 0
>
> So this "arbitrary" counter just went down from 100 to 096. I have the feeling that the current issues with this SSD are not really write problems, more hardware interface or firmware problems (as seen quite often for OCZ Vertex 2). Otherwise it would fail very early. According this data it should survive more 10th of years?
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: karl.wright@here.com [mailto:karl.wright@here.com]
>> Sent: Monday, August 19, 2013 5:07 PM
>> To: dev@lucene.apache.org
>> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
>>
>> I am told that SSD's are spec'd for only 70 full writes before they
>> get an error. The error block is set aside but eventually something
>> critical gets hit. So you should probably should expect this to happen again.
>>
>> Karl
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene tests killed one other SSD - Policeman Jenkins
Posted by Michael McCandless <lu...@mikemccandless.com>.
Er, which drives have write endurance only 70x? That's exceptionally
low? I thought this was more like ~1000x for modern consumer SSDs?
Mike McCandless
http://blog.mikemccandless.com
On Mon, Aug 19, 2013 at 12:35 PM, <ka...@here.com> wrote:
> " Only 70 full writes seems a little bit low for an SSD."
>
> That's what I thought. I was astounded to learn that that is in fact correct (at least for some of the drives we are using here). Automatic recovery is how the SSD copes with this failure rate.
>
> But it is entirely possible that the cause of your problems are something else.
>
> Karl
>
> -----Original Message-----
> From: ext Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, August 19, 2013 12:31 PM
> To: dev@lucene.apache.org
> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
>
> Hi Karl,
>
> Only 70 full writes seems a little bit low for an SSD. This SSD has a capacity of 64 Gigabytes so 53 Terabytes written is not bad for it. It already had the first few errors, so you might be right, as you see from this Smart data:
>
> 231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 0
>
> So this "arbitrary" counter just went down from 100 to 096. I have the feeling that the current issues with this SSD are not really write problems, more hardware interface or firmware problems (as seen quite often for OCZ Vertex 2). Otherwise it would fail very early. According this data it should survive more 10th of years?
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: karl.wright@here.com [mailto:karl.wright@here.com]
>> Sent: Monday, August 19, 2013 5:07 PM
>> To: dev@lucene.apache.org
>> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
>>
>> I am told that SSD's are spec'd for only 70 full writes before they
>> get an error. The error block is set aside but eventually something
>> critical gets hit. So you should probably should expect this to happen again.
>>
>> Karl
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: Lucene tests killed one other SSD - Policeman Jenkins
Posted by ka...@here.com.
" Only 70 full writes seems a little bit low for an SSD."
That's what I thought. I was astounded to learn that that is in fact correct (at least for some of the drives we are using here). Automatic recovery is how the SSD copes with this failure rate.
But it is entirely possible that the cause of your problems are something else.
Karl
-----Original Message-----
From: ext Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Monday, August 19, 2013 12:31 PM
To: dev@lucene.apache.org
Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
Hi Karl,
Only 70 full writes seems a little bit low for an SSD. This SSD has a capacity of 64 Gigabytes so 53 Terabytes written is not bad for it. It already had the first few errors, so you might be right, as you see from this Smart data:
231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 0
So this "arbitrary" counter just went down from 100 to 096. I have the feeling that the current issues with this SSD are not really write problems, more hardware interface or firmware problems (as seen quite often for OCZ Vertex 2). Otherwise it would fail very early. According this data it should survive more 10th of years?
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: karl.wright@here.com [mailto:karl.wright@here.com]
> Sent: Monday, August 19, 2013 5:07 PM
> To: dev@lucene.apache.org
> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
>
> I am told that SSD's are spec'd for only 70 full writes before they
> get an error. The error block is set aside but eventually something
> critical gets hit. So you should probably should expect this to happen again.
>
> Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org
RE: Lucene tests killed one other SSD - Policeman Jenkins
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Karl,
Only 70 full writes seems a little bit low for an SSD. This SSD has a capacity of 64 Gigabytes so 53 Terabytes written is not bad for it. It already had the first few errors, so you might be right, as you see from this Smart data:
231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 0
So this "arbitrary" counter just went down from 100 to 096. I have the feeling that the current issues with this SSD are not really write problems, more hardware interface or firmware problems (as seen quite often for OCZ Vertex 2). Otherwise it would fail very early. According this data it should survive more 10th of years?
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: karl.wright@here.com [mailto:karl.wright@here.com]
> Sent: Monday, August 19, 2013 5:07 PM
> To: dev@lucene.apache.org
> Subject: RE: Lucene tests killed one other SSD - Policeman Jenkins
>
> I am told that SSD's are spec'd for only 70 full writes before they get an error. The error
> block is set aside but eventually something critical gets hit. So you should probably should
> expect this to happen again.
>
> Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: Lucene tests killed one other SSD - Policeman Jenkins
Posted by ka...@here.com.
I am told that SSD's are spec'd for only 70 full writes before they get an error. The error block is set aside but eventually something critical gets hit. So you should probably should expect this to happen again.
Karl
-----Original Message-----
From: ext Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Monday, August 19, 2013 11:04 AM
To: dev@lucene.apache.org
Subject: Lucene tests killed one other SSD - Policeman Jenkins
Hi,
there were some problems with Policeman Jenkins the last days. The server died 6 times the last month, recently 2 times in 24 hours. After I moved away the swap file from the SSD, the failures were no longer fatal for the server but fatal for some Jenkins runs :-)
Finally the SSD device got unresponsible and only after a power cycle it was responsible again. The error messages in dmesg look similar to other dying OCX Vertex 2 drives.
Now the statistics: During the whole lifetime of this SSD (2.5 years; which is the lifetime of the server), it was mostly unused (it was just a "addon", provided by the hosting provider, thanks to Serverloft / Plusserver). 1.5 years ago, Robert Muir and also Mike McCandless decided to use the server of my own company SD DataSolutions to do more than idling most of the time: We installed Jenkins and 2 additional virtualbox machines on this server after the 2012 Lucene Revolution conference and the "spare" SSD was given as base for swap file, Jenkins Workspace and virtual disks for the Windows and Haskintosh machines.
During this time (1 year, 3 months) the SSD did hard work, according to SMART:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 112 112 050 Pre-fail Always - 0/61244435
5 Retired_Block_Count 0x0033 100 100 003 Pre-fail Always - 0
9 Power_On_Hours_and_Msec 0x0032 100 100 000 Old_age Always - 21904h+48m+22.180s
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 19
171 Program_Fail_Count 0x0032 000 000 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 000 000 000 Old_age Always - 0
174 Unexpect_Power_Loss_Ct 0x0030 000 000 000 Old_age Offline - 6
177 Wear_Range_Delta 0x0000 000 000 000 Old_age Offline - 2
181 Program_Fail_Count 0x0032 000 000 000 Old_age Always - 0
182 Erase_Fail_Count 0x0032 000 000 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
194 Temperature_Celsius 0x0022 001 129 000 Old_age Always - 1 (0 127 0 129)
195 ECC_Uncorr_Error_Count 0x001c 112 112 000 Old_age Offline - 0/61244435
196 Reallocated_Event_Count 0x0033 100 100 000 Pre-fail Always - 0
231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 0
233 SandForce_Internal 0x0000 000 000 000 Old_age Offline - 18752
234 SandForce_Internal 0x0032 000 000 000 Old_age Always - 53376
241 Lifetime_Writes_GiB 0x0032 000 000 000 Old_age Always - 53376
242 Lifetime_Reads_GiB 0x0032 000 000 000 Old_age Always - 22784
Last 2 lines are interesting:
53 Terabytes written to it and 22 Terabytes read from it. Ignore swap (mostly unused as swappiness is low), so our tests are reading and writing a lot!
And unfortunately after that it died (or almost died) this morning. Cause is unclear, it could also be broken SATA cable, but from the web the given error messages in "dmesg" seem to also be caused by drive failure (especially as it is a timeout, not DMA error)! See https://paste.apache.org/bjAH
So just to conclude: Lucene kills SSDs :-) Mike still has one Vertex 3 running (his Intel one died before).
Of course as this is a rented server, the hosting provider will replace the SSD (I was able to copy the data off, but the Jenkins workspace is not really important data, more the virtual machines). After that one more year with a new SSD, or should it survive longer? Let's see what type I will get as replacement. I have no idea when it is replaced, so excuse any jenkins downtime and after that maybe broken builds until all is settled again. At the moment Jenkins is running much slower from the RAID 1 harddisks (with lots of IOWAITS!).
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org
RE: Lucene tests killed one other SSD - Policeman Jenkins
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
the broken SSD was replaced yesterday evening. All seems fine. I moved the Jenkins workspace and (snapshots of the) virtual machine disks to the new SSD and game is going on!
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, August 19, 2013 5:04 PM
> To: dev@lucene.apache.org
> Subject: Lucene tests killed one other SSD - Policeman Jenkins
>
> Hi,
>
> there were some problems with Policeman Jenkins the last days. The server
> died 6 times the last month, recently 2 times in 24 hours. After I moved away
> the swap file from the SSD, the failures were no longer fatal for the server
> but fatal for some Jenkins runs :-)
>
> Finally the SSD device got unresponsible and only after a power cycle it was
> responsible again. The error messages in dmesg look similar to other dying
> OCX Vertex 2 drives.
>
> Now the statistics: During the whole lifetime of this SSD (2.5 years; which is
> the lifetime of the server), it was mostly unused (it was just a "addon",
> provided by the hosting provider, thanks to Serverloft / Plusserver). 1.5 years
> ago, Robert Muir and also Mike McCandless decided to use the server of my
> own company SD DataSolutions to do more than idling most of the time: We
> installed Jenkins and 2 additional virtualbox machines on this server after the
> 2012 Lucene Revolution conference and the "spare" SSD was given as base
> for swap file, Jenkins Workspace and virtual disks for the Windows and
> Haskintosh machines.
>
> During this time (1 year, 3 months) the SSD did hard work, according to
> SMART:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
> WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x000f 112 112 050 Pre-fail Always -
> 0/61244435
> 5 Retired_Block_Count 0x0033 100 100 003 Pre-fail Always - 0
> 9 Power_On_Hours_and_Msec 0x0032 100 100 000 Old_age Always
> - 21904h+48m+22.180s
> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always -
> 19
> 171 Program_Fail_Count 0x0032 000 000 000 Old_age Always - 0
> 172 Erase_Fail_Count 0x0032 000 000 000 Old_age Always - 0
> 174 Unexpect_Power_Loss_Ct 0x0030 000 000 000 Old_age Offline -
> 6
> 177 Wear_Range_Delta 0x0000 000 000 000 Old_age Offline - 2
> 181 Program_Fail_Count 0x0032 000 000 000 Old_age Always - 0
> 182 Erase_Fail_Count 0x0032 000 000 000 Old_age Always - 0
> 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
> 194 Temperature_Celsius 0x0022 001 129 000 Old_age Always -
> 1 (0 127 0 129)
> 195 ECC_Uncorr_Error_Count 0x001c 112 112 000 Old_age Offline -
> 0/61244435
> 196 Reallocated_Event_Count 0x0033 100 100 000 Pre-fail Always -
> 0
> 231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 0
> 233 SandForce_Internal 0x0000 000 000 000 Old_age Offline -
> 18752
> 234 SandForce_Internal 0x0032 000 000 000 Old_age Always -
> 53376
> 241 Lifetime_Writes_GiB 0x0032 000 000 000 Old_age Always -
> 53376
> 242 Lifetime_Reads_GiB 0x0032 000 000 000 Old_age Always -
> 22784
>
> Last 2 lines are interesting:
> 53 Terabytes written to it and 22 Terabytes read from it. Ignore swap (mostly
> unused as swappiness is low), so our tests are reading and writing a lot!
>
> And unfortunately after that it died (or almost died) this morning. Cause is
> unclear, it could also be broken SATA cable, but from the web the given error
> messages in "dmesg" seem to also be caused by drive failure (especially as it
> is a timeout, not DMA error)! See https://paste.apache.org/bjAH
>
> So just to conclude: Lucene kills SSDs :-) Mike still has one Vertex 3 running
> (his Intel one died before).
>
> Of course as this is a rented server, the hosting provider will replace the SSD
> (I was able to copy the data off, but the Jenkins workspace is not really
> important data, more the virtual machines). After that one more year with a
> new SSD, or should it survive longer? Let's see what type I will get as
> replacement. I have no idea when it is replaced, so excuse any jenkins
> downtime and after that maybe broken builds until all is settled again. At the
> moment Jenkins is running much slower from the RAID 1 harddisks (with lots
> of IOWAITS!).
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org