You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Bowen Song <bo...@bso.ng.INVALID> on 2021/02/09 13:34:02 UTC

Re: Cassandra on ZFS: disable compression?

I'm running Cassandra 3.11 on ZFS on Linux. I use the ZFS compression 
and have disabled the Cassandra SSTable compression.

I can't comment on the possible pros you've mentioned, because I didn't 
do enough tests on them. I know the compression ratio is marginally 
better with the same compression chunk size, but that's probably because 
the files containing the metadata are also compressed.

However, it's not trouble free. As Elliott said, this is a less tested 
code path. One problem I have ran into is the disk space check. 
Cassandra checks for available disk space before starting a compaction, 
and isn't aware of the ZFS compression. As the result of that, Cassandra 
often thinks there isn't enough free disk space left and refuse to 
compact the large SSTables.

My current workaround for the above issue is expensive. A large 
proportion of disk space is reserved (wasted) to avoid Cassandra running 
into this issue.

If you don't keep enough reserved disk space, you will eventually find 
one (or more) of your Cassandra nodes ends up in constant compaction 
loop, like this:

    WARN  [CompactionExecutor:7] 2021-02-04 19:50:33,636
    CompactionTask.java:94 - insufficient space to compact all requested
    files. 371463.6MB required,
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16048-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16049-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
    WARN  [CompactionExecutor:7] 2021-02-04 19:50:33,636
    CompactionTask.java:356 - Not enough space for compaction,
    371463.6MB estimated.  Reducing scope.
    WARN  [CompactionExecutor:7] 2021-02-04 19:50:33,636
    CompactionTask.java:94 - insufficient space to compact all requested
    files. 254743.16MB required,
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16048-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
    WARN  [CompactionExecutor:7] 2021-02-04 19:50:33,636
    CompactionTask.java:356 - Not enough space for compaction,
    254743.16MB estimated.  Reducing scope.
    WARN  [CompactionExecutor:7] 2021-02-04 19:50:33,637
    CompactionTask.java:94 - insufficient space to compact all requested
    files. 158297.66MB required,
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16048-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db')
    WARN  [CompactionExecutor:7] 2021-02-04 19:50:33,637
    CompactionTask.java:356 - Not enough space for compaction,
    158297.66MB estimated.  Reducing scope.
    WARN  [CompactionExecutor:7] 2021-02-04 20:09:05,607
    CompactionTask.java:94 - insufficient space to compact all requested
    files. 371463.6MB required,
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16049-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16050-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
    WARN  [CompactionExecutor:7] 2021-02-04 20:09:05,608
    CompactionTask.java:356 - Not enough space for compaction,
    371463.6MB estimated.  Reducing scope.
    WARN  [CompactionExecutor:7] 2021-02-04 20:09:05,608
    CompactionTask.java:94 - insufficient space to compact all requested
    files. 254743.16MB required,
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16050-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
    WARN  [CompactionExecutor:7] 2021-02-04 20:09:05,608
    CompactionTask.java:356 - Not enough space for compaction,
    254743.16MB estimated.  Reducing scope.
    WARN  [CompactionExecutor:7] 2021-02-04 20:09:05,608
    CompactionTask.java:94 - insufficient space to compact all requested
    files. 158297.66MB required,
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16050-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db')
    WARN  [CompactionExecutor:7] 2021-02-04 20:09:05,608
    CompactionTask.java:356 - Not enough space for compaction,
    158297.66MB estimated.  Reducing scope.
    WARN  [CompactionExecutor:7] 2021-02-04 20:27:28,482
    CompactionTask.java:94 - insufficient space to compact all requested
    files. 371463.6MB required,
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16049-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16052-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
    WARN  [CompactionExecutor:7] 2021-02-04 20:27:28,482
    CompactionTask.java:356 - Not enough space for compaction,
    371463.6MB estimated.  Reducing scope.
    WARN  [CompactionExecutor:7] 2021-02-04 20:27:28,482
    CompactionTask.java:94 - insufficient space to compact all requested
    files. 254743.16MB required,
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16052-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
    WARN  [CompactionExecutor:7] 2021-02-04 20:27:28,482
    CompactionTask.java:356 - Not enough space for compaction,
    254743.16MB estimated.  Reducing scope.
    WARN  [CompactionExecutor:7] 2021-02-04 20:27:28,482
    CompactionTask.java:94 - insufficient space to compact all requested
    files. 158297.66MB required,
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16052-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db')
    WARN  [CompactionExecutor:7] 2021-02-04 20:27:28,482
    CompactionTask.java:356 - Not enough space for compaction,
    158297.66MB estimated.  Reducing scope.
    WARN  [CompactionExecutor:7] 2021-02-04 20:45:56,505
    CompactionTask.java:94 - insufficient space to compact all requested
    files. 371463.6MB required,
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16049-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16053-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
    WARN  [CompactionExecutor:7] 2021-02-04 20:45:56,505
    CompactionTask.java:356 - Not enough space for compaction,
    371463.6MB estimated.  Reducing scope.
    WARN  [CompactionExecutor:7] 2021-02-04 20:45:56,505
    CompactionTask.java:94 - insufficient space to compact all requested
    files. 254743.16MB required,
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16053-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
    WARN  [CompactionExecutor:7] 2021-02-04 20:45:56,505
    CompactionTask.java:356 - Not enough space for compaction,
    254743.16MB estimated.  Reducing scope.
    WARN  [CompactionExecutor:7] 2021-02-04 20:45:56,505
    CompactionTask.java:94 - insufficient space to compact all requested
    files. 158297.66MB required,
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16053-big-Data.db'),
    BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db')
    WARN  [CompactionExecutor:7] 2021-02-04 20:45:56,505
    CompactionTask.java:356 - Not enough space for compaction,
    158297.66MB estimated.  Reducing scope.

and keep going on infinitely, until someone notices it (hopefully 
because of an alert, not an outage) and gets it fixed.


On 25/01/2021 15:06, Lapo Luchini wrote:
> Hi,
>     I'm using a fairly standard install of Cassandra 3.11 on FreeBSD 
> 12, by default filesystem is compressed using LZ4 and Cassandra tables 
> are compressed using LZ4 as well.
>
> I was wondering if anybody had data about this already (or else, I 
> will probably do some tests myself, eventually): would it be a nice 
> idea to disable Cassandra compression and rely only on ZFS one?
>
> In principle I can see some pros:
> 1. it's done in kernel, might be slightly faster
> 2. can (probably) compress more data, as I see a 1.02 compression factor
>    on filesystem even if I have compressed data in tables already
> 3. in upcoming ZFS version I will be able to use Zstd compression
>    (probably before Cassandra 4.0 is gold)
> 4. (can inspect  compression directly at filesystem level)
>
> But on the other hand application-level compression could have its 
> advantages.
>
> cheers,
>