You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Ekaterina Dimitrova (Jira)" <ji...@apache.org> on 2021/02/12 22:09:00 UTC
[jira] [Comment Edited] (CASSANDRA-16441) Snapshot_test.py DTests
failing
[ https://issues.apache.org/jira/browse/CASSANDRA-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284014#comment-17284014 ]
Ekaterina Dimitrova edited comment on CASSANDRA-16441 at 2/12/21, 10:08 PM:
----------------------------------------------------------------------------
I wasn't able to reproduce it on my own in any way.
But from the CircleCI log it seems as a legit issue. A node is not able to be brought up because of the following issue:
{code:java}
INFO [main] 2021-02-11 17:25:08,125 StorageService.java:830 - Populating token metadata from system tables
INFO [main] 2021-02-11 17:25:08,132 StorageService.java:824 - Token metadata:
ERROR [main] 2021-02-11 17:25:08,153 SchemaKeyspace.java:942 - No partition columns found for table system_auth.network_permissions in system_schema.columns. This may be due to corruption or concurrent dropping and altering of a table. If this table is supposed to be dropped, restart cassandra with -Dcassandra.ignore_corrupted_schema_tables=true and run the following query to cleanup: "DELETE FROM system_schema.tables WHERE keyspace_name = 'system_auth' AND table_name = 'network_permissions'; DELETE FROM system_schema.columns WHERE keyspace_name = 'system_auth' AND table_name = 'network_permissions';" If the table is not supposed to be dropped, restore system_schema.columns sstables from backups.
ERROR [main] 2021-02-11 17:25:08,155 CassandraDaemon.java:271 - Error while loading schema:
org.apache.cassandra.schema.SchemaKeyspace$MissingColumns: Columns not found in schema table for system_auth.network_permissions
at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:998)
at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:962)
at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:921)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:880)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:871)
at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:859)
at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:100)
at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:89)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:267)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:676)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:800)
ERROR [main] 2021-02-11 17:25:08,155 CassandraDaemon.java:822 - Exception encountered during startup
org.apache.cassandra.schema.SchemaKeyspace$MissingColumns: Columns not found in schema table for system_auth.network_permissions
at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:998)
at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:962)
at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:921)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:880)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:871)
at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:859)
at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:100)
at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:89)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:267)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:676)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:800){code}
I just started a multiplexer job in a Jenkins environment.
Considering it is flaky, it seems like some race condition maybe? I didn't have a lot of time to dig into the log/code for more details today but felt like leaving here what I have as I will be out almost half of the next week so in case someone is interested and has time to also look at it in the meanwhile.
was (Author: e.dimitrova):
I wasn't able to reproduce it on my own in any way.
But from the CircleCI log it seems as a legit issue. A node is not able to be brought up because of the following issue:
{code:java}
INFO [main] 2021-02-11 17:25:08,125 StorageService.java:830 - Populating token metadata from system tables
INFO [main] 2021-02-11 17:25:08,132 StorageService.java:824 - Token metadata:
ERROR [main] 2021-02-11 17:25:08,153 SchemaKeyspace.java:942 - No partition columns found for table system_auth.network_permissions in system_schema.columns. This may be due to corruption or concurrent dropping and altering of a table. If this table is supposed to be dropped, restart cassandra with -Dcassandra.ignore_corrupted_schema_tables=true and run the following query to cleanup: "DELETE FROM system_schema.tables WHERE keyspace_name = 'system_auth' AND table_name = 'network_permissions'; DELETE FROM system_schema.columns WHERE keyspace_name = 'system_auth' AND table_name = 'network_permissions';" If the table is not supposed to be dropped, restore system_schema.columns sstables from backups.
ERROR [main] 2021-02-11 17:25:08,155 CassandraDaemon.java:271 - Error while loading schema:
org.apache.cassandra.schema.SchemaKeyspace$MissingColumns: Columns not found in schema table for system_auth.network_permissions
at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:998)
at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:962)
at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:921)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:880)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:871)
at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:859)
at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:100)
at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:89)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:267)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:676)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:800)
ERROR [main] 2021-02-11 17:25:08,155 CassandraDaemon.java:822 - Exception encountered during startup
org.apache.cassandra.schema.SchemaKeyspace$MissingColumns: Columns not found in schema table for system_auth.network_permissions
at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:998)
at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:962)
at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:921)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:880)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:871)
at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:859)
at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:100)
at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:89)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:267)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:676)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:800){code}
I just started a multiplexer job in a Jenkins environment.
Considering it is flaky, it seems like some race condition maybe? I didn't have a lot of time to dig into the log/code for more details today but felt like leaving here what I have as I will be out almost half next week so in case someone is interested and has time to look at it too in the meanwhile.
> Snapshot_test.py DTests failing
> -------------------------------
>
> Key: CASSANDRA-16441
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16441
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest/python
> Reporter: Ekaterina Dimitrova
> Priority: Normal
> Fix For: 4.0-beta5, 4.0-beta
>
>
> [https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/623/workflows/1c89bd58-bcd1-45dc-b923-3aba245a78a6/jobs/3451]
> These failures are also observed for other patches in Jenkins dev (weird but they are not seen failing in the latest builds and they are failing with completely unrelated different patches...)
> [https://jenkins-cm4.apache.org/job/Cassandra-devbranch/375/#showFailuresLink]
> [https://jenkins-cm4.apache.org/job/Cassandra-devbranch/368/]
> Jenkins dev history for one of the tests:
> https://jenkins-cm4.apache.org/job/Cassandra-devbranch/375/testReport/junit/dtest-novnode.snapshot_test/TestArchiveCommitlog/test_archive_commitlog_with_active_commitlog/history/
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org