Ext4 data corruption in stable kernels

[Jon Haddad <jo...@jonhaddad.com>]

Hey folks,

Just wanted to raise awareness about a I/O issue that seems to be affecting
some Linux Kernal releases that were listed as STABLE, causing corruption
when using the ext4 filesystem with direct I/O.  I don't have time to get a
great understanding of the full scope of the issue, what versions are
affected, etc, I just want to get this in front of the project.  I am
disappointed that this might negatively affect our ability to leverage
direct I/O for both the commitlog (recently merged) and SSTables
(potentially a future use case), since users won't be able to discern
between a bug we ship and one that we hit as a result of our filesystem
choices.

I think it might be worth putting a note in our docs and in the config to
warn the user to ensure they're not affected, and we may even want to
consider hiding this feature if the blast radius is significant enough that
users would be affected.

https://lwn.net/Articles/954285/

Jon

Re: Ext4 data corruption in stable kernels

[Jacek Lewandowski <le...@gmail.com>]

Frankly there only two kernel versions mentioned there. I've created
https://issues.apache.org/jira/browse/CASSANDRA-19196 to do something with
that.


pon., 11 gru 2023 o 21:05 Jon Haddad <jo...@jonhaddad.com> napisał(a):

> Like I said, I didn't have time to verify the full scope and what's
> affected, just that some stable kernels are affected.  Adding to the
> problem is that it might be vendor specific as well.  For example, RH might
> backport an upstream patch in the kernel they ship that's non-standard.
>
> Hopefully someone compiles a list.
>
> Jon
>
> On Mon, Dec 11, 2023 at 11:51 AM Jacek Lewandowski <
> lewandowski.jacek@gmail.com> wrote:
>
>> Aren't only specific kernels affected? If we can detect the kernel
>> version, the feature can be force disabled with the problematic kernels
>>
>>
>> pon., 11 gru 2023, 20:45 użytkownik Jon Haddad <jo...@jonhaddad.com>
>> napisał:
>>
>>> Hey folks,
>>>
>>> Just wanted to raise awareness about a I/O issue that seems to be
>>> affecting some Linux Kernal releases that were listed as STABLE, causing
>>> corruption when using the ext4 filesystem with direct I/O.  I don't have
>>> time to get a great understanding of the full scope of the issue, what
>>> versions are affected, etc, I just want to get this in front of the
>>> project.  I am disappointed that this might negatively affect our ability
>>> to leverage direct I/O for both the commitlog (recently merged) and
>>> SSTables (potentially a future use case), since users won't be able to
>>> discern between a bug we ship and one that we hit as a result of our
>>> filesystem choices.
>>>
>>> I think it might be worth putting a note in our docs and in the config
>>> to warn the user to ensure they're not affected, and we may even want to
>>> consider hiding this feature if the blast radius is significant enough that
>>> users would be affected.
>>>
>>> https://lwn.net/Articles/954285/
>>>
>>> Jon
>>>
>>

Re: Ext4 data corruption in stable kernels

[Jon Haddad <jo...@jonhaddad.com>]

Like I said, I didn't have time to verify the full scope and what's
affected, just that some stable kernels are affected.  Adding to the
problem is that it might be vendor specific as well.  For example, RH might
backport an upstream patch in the kernel they ship that's non-standard.

Hopefully someone compiles a list.

Jon

On Mon, Dec 11, 2023 at 11:51 AM Jacek Lewandowski <
lewandowski.jacek@gmail.com> wrote:

> Aren't only specific kernels affected? If we can detect the kernel
> version, the feature can be force disabled with the problematic kernels
>
>
> pon., 11 gru 2023, 20:45 użytkownik Jon Haddad <jo...@jonhaddad.com>
> napisał:
>
>> Hey folks,
>>
>> Just wanted to raise awareness about a I/O issue that seems to be
>> affecting some Linux Kernal releases that were listed as STABLE, causing
>> corruption when using the ext4 filesystem with direct I/O.  I don't have
>> time to get a great understanding of the full scope of the issue, what
>> versions are affected, etc, I just want to get this in front of the
>> project.  I am disappointed that this might negatively affect our ability
>> to leverage direct I/O for both the commitlog (recently merged) and
>> SSTables (potentially a future use case), since users won't be able to
>> discern between a bug we ship and one that we hit as a result of our
>> filesystem choices.
>>
>> I think it might be worth putting a note in our docs and in the config to
>> warn the user to ensure they're not affected, and we may even want to
>> consider hiding this feature if the blast radius is significant enough that
>> users would be affected.
>>
>> https://lwn.net/Articles/954285/
>>
>> Jon
>>
>

Re: Ext4 data corruption in stable kernels

[Jacek Lewandowski <le...@gmail.com>]

Aren't only specific kernels affected? If we can detect the kernel version,
the feature can be force disabled with the problematic kernels


pon., 11 gru 2023, 20:45 użytkownik Jon Haddad <jo...@jonhaddad.com> napisał:

> Hey folks,
>
> Just wanted to raise awareness about a I/O issue that seems to be
> affecting some Linux Kernal releases that were listed as STABLE, causing
> corruption when using the ext4 filesystem with direct I/O.  I don't have
> time to get a great understanding of the full scope of the issue, what
> versions are affected, etc, I just want to get this in front of the
> project.  I am disappointed that this might negatively affect our ability
> to leverage direct I/O for both the commitlog (recently merged) and
> SSTables (potentially a future use case), since users won't be able to
> discern between a bug we ship and one that we hit as a result of our
> filesystem choices.
>
> I think it might be worth putting a note in our docs and in the config to
> warn the user to ensure they're not affected, and we may even want to
> consider hiding this feature if the blast radius is significant enough that
> users would be affected.
>
> https://lwn.net/Articles/954285/
>
> Jon
>