You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Luciano Greiner <lu...@gmail.com> on 2025/01/30 02:38:12 UTC
Cassandra 5 upgrade and Schema Agreement failures
Hello.
I am upgrading a small Cassandra 4.1.3 cluster (2 sites, 3 nodes each)
to Cassandra 5.
Given we're using an old Centos OS in our nodes, I decided to get new
nodes provisioned in the cluster (with version 5 on compatibility
mode), and then decommission the old nodes when all is completed.
There is an issue about schema agreement when I try to bootstrap the
first Cassandra 5 node into the cluster:
WARN [main] 2025-01-30 01:55:40,012
DefaultSchemaUpdateHandler.java:131 - There are nodes in the cluster
with a different schema version than us, from which we did not merge
schemas: our version: (59adb24e-f3cd-3e02-97f0-5b395827453f),
outstanding versions -> endpoints:
{9bd8dc68-76f1-3b55-b60a-bb69ea16e53a=[/10.1.9.10:7000,
/10.1.9.11:7000, /10.2.8.20:7000, /10.1.9.13:7000, /10.2.8.24:7000,
/10.2.8.25:7000]}. Use -Dcassandra.skip_schema_check=true to ignore
this, -Dcassandra.skip_schema_check_for_endpoints=<ep1[,epN]> to skip
specific endpoints, or
-Dcassandra.skip_schema_check_for_versions=<ver1[,verN]> to skip
specific schema versions
ERROR [main] 2025-01-30 01:55:40,012
DefaultSchemaUpdateHandler.java:142 - Didn't receive schemas for all
known versions within the PT30S. Use
-Dcassandra.skip_schema_check=true to skip this check.
ERROR [main] 2025-01-30 01:55:40,014 CassandraDaemon.java:887 -
Exception encountered during startup
java.lang.IllegalStateException: Could not achieve schema readiness in PT30S
So I ran the describecluster. All 6 nodes are using the same schema
version 9bd8dc68-76f1-3b55-b60a-bb69ea16e53a. Still, I've gone ahead
and ran resetlocalschema and restarted each node, one at a time. No
luck.
Then I tried to skip the schema check, but then I get several table
not found errors raised on the new node logs until it eventually shuts
down.
I am running out of ideas here.
Does anybody have any thoughts?
Thank you
Luciano
Re: Cassandra 5 upgrade and Schema Agreement failures
Posted by Paul Chandler <pa...@redshots.com>.
Yes, I would add the 6 new nodes, then decommission the 6 original nodes, then upgrade to 5.0.
Remember to change the seed node values before you decommission the old nodes.
Thanks
Paul
> On 30 Jan 2025, at 14:42, Luciano Greiner <lu...@gmail.com> wrote:
>
> Ok, so you mean I setup these new 6 nodes as 4.1.3, then just upgrade
> the software, correct?
>
> Thank you!
>
> Luciano
>
>
> On Thu, Jan 30, 2025 at 6:57 AM Paul Chandler <pa...@redshots.com> wrote:
>>
>> Hi Luciano,
>>
>> The problem occurs due to Cassandra 5 making changes to the system tables, so the cluster will be in schema mismatch during the upgrade process, until all the nodes are on 5.0
>>
>> Normally this would not be a problem, as the system tables are not replicated anyway, but, as you are finding, it will stop you adding, or removing nodes during an upgrade process.
>>
>> Given this, it maybe best to split this operation, firstly to provision the new nodes with 4.1.3, then upgrade the nodes to Cassandra 5.
>>
>> Thanks
>>
>> Paul
>>
>>
>>
>>> On 30 Jan 2025, at 02:38, Luciano Greiner <lu...@gmail.com> wrote:
>>>
>>> Hello.
>>>
>>> I am upgrading a small Cassandra 4.1.3 cluster (2 sites, 3 nodes each)
>>> to Cassandra 5.
>>>
>>> Given we're using an old Centos OS in our nodes, I decided to get new
>>> nodes provisioned in the cluster (with version 5 on compatibility
>>> mode), and then decommission the old nodes when all is completed.
>>>
>>> There is an issue about schema agreement when I try to bootstrap the
>>> first Cassandra 5 node into the cluster:
>>>
>>> WARN [main] 2025-01-30 01:55:40,012
>>> DefaultSchemaUpdateHandler.java:131 - There are nodes in the cluster
>>> with a different schema version than us, from which we did not merge
>>> schemas: our version: (59adb24e-f3cd-3e02-97f0-5b395827453f),
>>> outstanding versions -> endpoints:
>>> {9bd8dc68-76f1-3b55-b60a-bb69ea16e53a=[/10.1.9.10:7000,
>>> /10.1.9.11:7000, /10.2.8.20:7000, /10.1.9.13:7000, /10.2.8.24:7000,
>>> /10.2.8.25:7000]}. Use -Dcassandra.skip_schema_check=true to ignore
>>> this, -Dcassandra.skip_schema_check_for_endpoints=<ep1[,epN]> to skip
>>> specific endpoints, or
>>> -Dcassandra.skip_schema_check_for_versions=<ver1[,verN]> to skip
>>> specific schema versions
>>> ERROR [main] 2025-01-30 01:55:40,012
>>> DefaultSchemaUpdateHandler.java:142 - Didn't receive schemas for all
>>> known versions within the PT30S. Use
>>> -Dcassandra.skip_schema_check=true to skip this check.
>>> ERROR [main] 2025-01-30 01:55:40,014 CassandraDaemon.java:887 -
>>> Exception encountered during startup
>>> java.lang.IllegalStateException: Could not achieve schema readiness in PT30S
>>>
>>> So I ran the describecluster. All 6 nodes are using the same schema
>>> version 9bd8dc68-76f1-3b55-b60a-bb69ea16e53a. Still, I've gone ahead
>>> and ran resetlocalschema and restarted each node, one at a time. No
>>> luck.
>>>
>>> Then I tried to skip the schema check, but then I get several table
>>> not found errors raised on the new node logs until it eventually shuts
>>> down.
>>>
>>> I am running out of ideas here.
>>>
>>> Does anybody have any thoughts?
>>>
>>> Thank you
>>>
>>> Luciano
>>
Re: Cassandra 5 upgrade and Schema Agreement failures
Posted by Luciano Greiner <lu...@gmail.com>.
Ok, so you mean I setup these new 6 nodes as 4.1.3, then just upgrade
the software, correct?
Thank you!
Luciano
On Thu, Jan 30, 2025 at 6:57 AM Paul Chandler <pa...@redshots.com> wrote:
>
> Hi Luciano,
>
> The problem occurs due to Cassandra 5 making changes to the system tables, so the cluster will be in schema mismatch during the upgrade process, until all the nodes are on 5.0
>
> Normally this would not be a problem, as the system tables are not replicated anyway, but, as you are finding, it will stop you adding, or removing nodes during an upgrade process.
>
> Given this, it maybe best to split this operation, firstly to provision the new nodes with 4.1.3, then upgrade the nodes to Cassandra 5.
>
> Thanks
>
> Paul
>
>
>
> > On 30 Jan 2025, at 02:38, Luciano Greiner <lu...@gmail.com> wrote:
> >
> > Hello.
> >
> > I am upgrading a small Cassandra 4.1.3 cluster (2 sites, 3 nodes each)
> > to Cassandra 5.
> >
> > Given we're using an old Centos OS in our nodes, I decided to get new
> > nodes provisioned in the cluster (with version 5 on compatibility
> > mode), and then decommission the old nodes when all is completed.
> >
> > There is an issue about schema agreement when I try to bootstrap the
> > first Cassandra 5 node into the cluster:
> >
> > WARN [main] 2025-01-30 01:55:40,012
> > DefaultSchemaUpdateHandler.java:131 - There are nodes in the cluster
> > with a different schema version than us, from which we did not merge
> > schemas: our version: (59adb24e-f3cd-3e02-97f0-5b395827453f),
> > outstanding versions -> endpoints:
> > {9bd8dc68-76f1-3b55-b60a-bb69ea16e53a=[/10.1.9.10:7000,
> > /10.1.9.11:7000, /10.2.8.20:7000, /10.1.9.13:7000, /10.2.8.24:7000,
> > /10.2.8.25:7000]}. Use -Dcassandra.skip_schema_check=true to ignore
> > this, -Dcassandra.skip_schema_check_for_endpoints=<ep1[,epN]> to skip
> > specific endpoints, or
> > -Dcassandra.skip_schema_check_for_versions=<ver1[,verN]> to skip
> > specific schema versions
> > ERROR [main] 2025-01-30 01:55:40,012
> > DefaultSchemaUpdateHandler.java:142 - Didn't receive schemas for all
> > known versions within the PT30S. Use
> > -Dcassandra.skip_schema_check=true to skip this check.
> > ERROR [main] 2025-01-30 01:55:40,014 CassandraDaemon.java:887 -
> > Exception encountered during startup
> > java.lang.IllegalStateException: Could not achieve schema readiness in PT30S
> >
> > So I ran the describecluster. All 6 nodes are using the same schema
> > version 9bd8dc68-76f1-3b55-b60a-bb69ea16e53a. Still, I've gone ahead
> > and ran resetlocalschema and restarted each node, one at a time. No
> > luck.
> >
> > Then I tried to skip the schema check, but then I get several table
> > not found errors raised on the new node logs until it eventually shuts
> > down.
> >
> > I am running out of ideas here.
> >
> > Does anybody have any thoughts?
> >
> > Thank you
> >
> > Luciano
>
Re: Cassandra 5 upgrade and Schema Agreement failures
Posted by Paul Chandler <pa...@redshots.com>.
Hi Luciano,
The problem occurs due to Cassandra 5 making changes to the system tables, so the cluster will be in schema mismatch during the upgrade process, until all the nodes are on 5.0
Normally this would not be a problem, as the system tables are not replicated anyway, but, as you are finding, it will stop you adding, or removing nodes during an upgrade process.
Given this, it maybe best to split this operation, firstly to provision the new nodes with 4.1.3, then upgrade the nodes to Cassandra 5.
Thanks
Paul
> On 30 Jan 2025, at 02:38, Luciano Greiner <lu...@gmail.com> wrote:
>
> Hello.
>
> I am upgrading a small Cassandra 4.1.3 cluster (2 sites, 3 nodes each)
> to Cassandra 5.
>
> Given we're using an old Centos OS in our nodes, I decided to get new
> nodes provisioned in the cluster (with version 5 on compatibility
> mode), and then decommission the old nodes when all is completed.
>
> There is an issue about schema agreement when I try to bootstrap the
> first Cassandra 5 node into the cluster:
>
> WARN [main] 2025-01-30 01:55:40,012
> DefaultSchemaUpdateHandler.java:131 - There are nodes in the cluster
> with a different schema version than us, from which we did not merge
> schemas: our version: (59adb24e-f3cd-3e02-97f0-5b395827453f),
> outstanding versions -> endpoints:
> {9bd8dc68-76f1-3b55-b60a-bb69ea16e53a=[/10.1.9.10:7000,
> /10.1.9.11:7000, /10.2.8.20:7000, /10.1.9.13:7000, /10.2.8.24:7000,
> /10.2.8.25:7000]}. Use -Dcassandra.skip_schema_check=true to ignore
> this, -Dcassandra.skip_schema_check_for_endpoints=<ep1[,epN]> to skip
> specific endpoints, or
> -Dcassandra.skip_schema_check_for_versions=<ver1[,verN]> to skip
> specific schema versions
> ERROR [main] 2025-01-30 01:55:40,012
> DefaultSchemaUpdateHandler.java:142 - Didn't receive schemas for all
> known versions within the PT30S. Use
> -Dcassandra.skip_schema_check=true to skip this check.
> ERROR [main] 2025-01-30 01:55:40,014 CassandraDaemon.java:887 -
> Exception encountered during startup
> java.lang.IllegalStateException: Could not achieve schema readiness in PT30S
>
> So I ran the describecluster. All 6 nodes are using the same schema
> version 9bd8dc68-76f1-3b55-b60a-bb69ea16e53a. Still, I've gone ahead
> and ran resetlocalschema and restarted each node, one at a time. No
> luck.
>
> Then I tried to skip the schema check, but then I get several table
> not found errors raised on the new node logs until it eventually shuts
> down.
>
> I am running out of ideas here.
>
> Does anybody have any thoughts?
>
> Thank you
>
> Luciano