SSD fatigue

[Phil H <gi...@gmail.com>]

Hi there,

I have been reading some reports of people having significant SSD degredation after 100+Tb of writes, due to the chemistry of NAND gates (or something?!).  Based on my current NiFi disk I/O, I think that works out to less than 12 months of usage.

What are other users with large scale NiFi systems thinking/doing about this?

Thanks,
Phil

Re: SSD fatigue

[Arpad Boda <ab...@hortonworks.com>]

Just some details to add here:
-Make sure TRIM is enabled, it extends lifetime of your SSDs.
-The reason larger SSDs tend to last longer is simple: firmware load-balances usage between cells, higher capacity means more cells -> less frequent data modification in one cell. 
-As Michael mentioned, SSDs are different. I would suggest checking the differences between SLC, MLC and TLC and I personally wouldn't recommend TLC SSDs for such usage. SLCs tend to be tremendously expensive, so go for MLC. 

I've found this article about the lifetime of SSDs:
https://techreport.com/review/26523/the-ssd-endurance-experiment-casualties-on-the-way-to-a-petabyte

It seems that even the less endurable ones lasted for at least 500TBs of data written, which - in my opinion - is more than enough. 

Regards,
Arpad

On 14/12/2018, 23:01, "Michael Moser" <mo...@gmail.com> wrote:

    Hello Phil,
    
    When supporting an I/O intensive application such as NiFi at the enterprise
    level, you have to plan for failures in your storage layer.  Disks are
    going to fail and you only get to choose how that failure will affect your
    system.  Here are some things to think about.
    
    + Use RAID for data redundancy, to mitigate possible data loss when a disk
    fails.  Buy spares.
    + Buy larger SSDs because they have greater reserve capacity and larger TBW
    endurance.
    + Not all SSDs are created equal.  Some SSDs are cheaper with less
    endurance on purpose.  "You get what you pay for."
    + Use SMART to monitor your SSDs.  They will tell you when their end of
    life is near.
    
    Hope this helps,
    -- Mike
    
    
    On Thu, Dec 13, 2018 at 4:05 AM Phil H <gi...@gmail.com> wrote:
    
    > Hi there,
    >
    > I have been reading some reports of people having significant SSD
    > degredation after 100+Tb of writes, due to the chemistry of NAND gates (or
    > something?!).  Based on my current NiFi disk I/O, I think that works out to
    > less than 12 months of usage.
    >
    > What are other users with large scale NiFi systems thinking/doing about
    > this?
    >
    > Thanks,
    > Phil
    >
    

Re: SSD fatigue

[Michael Moser <mo...@gmail.com>]

Hello Phil,

When supporting an I/O intensive application such as NiFi at the enterprise
level, you have to plan for failures in your storage layer.  Disks are
going to fail and you only get to choose how that failure will affect your
system.  Here are some things to think about.

+ Use RAID for data redundancy, to mitigate possible data loss when a disk
fails.  Buy spares.
+ Buy larger SSDs because they have greater reserve capacity and larger TBW
endurance.
+ Not all SSDs are created equal.  Some SSDs are cheaper with less
endurance on purpose.  "You get what you pay for."
+ Use SMART to monitor your SSDs.  They will tell you when their end of
life is near.

Hope this helps,
-- Mike


On Thu, Dec 13, 2018 at 4:05 AM Phil H <gi...@gmail.com> wrote:

> Hi there,
>
> I have been reading some reports of people having significant SSD
> degredation after 100+Tb of writes, due to the chemistry of NAND gates (or
> something?!).  Based on my current NiFi disk I/O, I think that works out to
> less than 12 months of usage.
>
> What are other users with large scale NiFi systems thinking/doing about
> this?
>
> Thanks,
> Phil
>