You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Oleksandr Shulgin <ol...@zalando.de> on 2018/01/24 09:40:59 UTC
Upgrading sstables not using all available compaction slots on
version 2.2
Hello,
In the process of upgrading our cluster from 2.1 to 2.2 we have triggered
the SSTable rewriting process like this:
$ nodetool upgradesstables -j 4 # concurrent_compactors=5
Then if we would immediately check the compactionstats, we see that 4
compactions of type 'Upgrade sstables' are running for the same
keyspace/table. Fine.
What is surprising, though, is that when any of the tasks which are smaller
in size complete, no new tasks are added and we are left waiting for a
smaller number of long running tasks to complete, e.g:
pending tasks: 4
id compaction type keyspace
table completed total unit progress
d535e962-00de-11e8-9c2c-ade7b0b922e3 Upgrade sstables cel_data
event_history_unbounded 32.87 GB 84.12 GB bytes 39.08%
d535e960-00de-11e8-9c2c-ade7b0b922e3 Upgrade sstables cel_data
event_history_unbounded 33.15 GB 84.79 GB bytes 39.09%
Periodically, an ordinary compaction would be listed, but no new Upgrade
tasks appear.
Only after all of the "current" tasks are finished, the new 4 tasks are
scheduled and the process repeats.
We would expect that new tasks would be scheduled as soon as a compaction
slot is freed up, but this doesn't seem to happen. Is this by design? I
do not see any open issue about this in Jira.
Cheers,
--
Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
127-59-707
Re: Upgrading sstables not using all available compaction slots on
version 2.2
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Thu, Feb 1, 2018 at 9:23 AM, Oleksandr Shulgin <
oleksandr.shulgin@zalando.de> wrote:
> On 1 Feb 2018 06:51, "kurt greaves" <ku...@instaclustr.com> wrote:
>
> Would you be able to create a JIRA ticket for this? Not sure if this is
> still a problem in 3.0+ but worth creating a ticket to investigate. It'd be
> really helpful if you could try and reproduce on 3.0.15 or 3.11.1 to see if
> it's an issue there as well.
>
>
> Our next planned step is to upgrade to 3.0.15, so we will see pretty soon
> if that's still an issue in the newer version.
>
Still the same issue with 3.0. I've opened a ticket:
https://issues.apache.org/jira/browse/CASSANDRA-14210
--
Alex
Re: Upgrading sstables not using all available compaction slots on
version 2.2
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On 1 Feb 2018 06:51, "kurt greaves" <ku...@instaclustr.com> wrote:
Would you be able to create a JIRA ticket for this? Not sure if this is
still a problem in 3.0+ but worth creating a ticket to investigate. It'd be
really helpful if you could try and reproduce on 3.0.15 or 3.11.1 to see if
it's an issue there as well.
Our next planned step is to upgrade to 3.0.15, so we will see pretty soon
if that's still an issue in the newer version.
--
Alex
Re: Upgrading sstables not using all available compaction slots on
version 2.2
Posted by kurt greaves <ku...@instaclustr.com>.
Would you be able to create a JIRA ticket for this? Not sure if this is
still a problem in 3.0+ but worth creating a ticket to investigate. It'd be
really helpful if you could try and reproduce on 3.0.15 or 3.11.1 to see if
it's an issue there as well.
Re: Upgrading sstables not using all available compaction slots on
version 2.2
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Wed, Jan 24, 2018 at 10:40 AM, Oleksandr Shulgin <
oleksandr.shulgin@zalando.de> wrote:
> Hello,
>
> In the process of upgrading our cluster from 2.1 to 2.2 we have triggered
> the SSTable rewriting process like this:
>
> $ nodetool upgradesstables -j 4 # concurrent_compactors=5
>
> Then if we would immediately check the compactionstats, we see that 4
> compactions of type 'Upgrade sstables' are running for the same
> keyspace/table. Fine.
>
> What is surprising, though, is that when any of the tasks which are
> smaller in size complete, no new tasks are added and we are left waiting
> for a smaller number of long running tasks to complete, e.g:
>
> pending tasks: 4
> id compaction type keyspace
> table completed total unit progress
> d535e962-00de-11e8-9c2c-ade7b0b922e3 Upgrade sstables cel_data
> event_history_unbounded 32.87 GB 84.12 GB bytes 39.08%
> d535e960-00de-11e8-9c2c-ade7b0b922e3 Upgrade sstables cel_data
> event_history_unbounded 33.15 GB 84.79 GB bytes 39.09%
>
> Periodically, an ordinary compaction would be listed, but no new Upgrade
> tasks appear.
> Only after all of the "current" tasks are finished, the new 4 tasks are
> scheduled and the process repeats.
>
> We would expect that new tasks would be scheduled as soon as a compaction
> slot is freed up, but this doesn't seem to happen. Is this by design? I
> do not see any open issue about this in Jira.
>
FYI: Because of this issue it has taken more than 7 days instead of the
estimated 1.5-2 days, based on the available throughput, to migrate ~50 TiB
of data files spread over 12 nodes.
--
Alex