multiple table directories for system_schema keyspace

[John Sanda <jo...@gmail.com>]

On a couple different occasions I have run into this exception at start up:

Exception (org.apache.cassandra.exceptions.InvalidRequestException)
encountered during startup: Unknown type <user type>
org.apache.cassandra.exceptions.InvalidRequestException: Unknown type <user
type>
        at
org.apache.cassandra.cql3.CQL3Type$Raw$RawUT.prepare(CQL3Type.java:745)
        at
org.apache.cassandra.cql3.CQL3Type$Raw.prepareInternal(CQL3Type.java:533)
        at
org.apache.cassandra.schema.CQLTypeParser.parse(CQLTypeParser.java:53)
        at
org.apache.cassandra.schema.SchemaKeyspace.createColumnFromRow(SchemaKeyspace.java:1052)
        at
org.apache.cassandra.schema.SchemaKeyspace.lambda$fetchColumns$12(SchemaKeyspace.java:1038)

This was with Cassandra 3.0.12 running in Kubernetes, which means that IP
address changes for the Cassandra node can and will happen. Nowhere in
client code does the UDT get dropped. I came across
https://issues.apache.org/jira/browse/CASSANDRA-13739 which got me
wondering if this particular Cassandra node wound up with another version
of the system_schema.types table which did not have the UDT.

In what circumstances could I end up with multiple table directories for
the tables in system_schema? Right now I am just guessing that I wound up
with a newer (or different) version of the system_schema.types table.
Unfortunately, I no longer have access to the environment to confirm/deny
what was happening. I just want to better understand so I can avoid it in
the future.


- John

Re: multiple table directories for system_schema keyspace

[Rahul Singh <ra...@gmail.com>]

Happens to any keyspace — not just system — if there are competing processes initializing the system , creating / altering new things without CL=all it may do this. I ran into a scenario where when permissions were flipped to a non Cassandra user, the Cassandra daemon lost access to the data so it reinitialized the system.

--
Rahul Singh
rahul.singh@anant.us

Anant Corporation

On Apr 17, 2018, 2:25 PM -0500, John Sanda <jo...@gmail.com>, wrote:
> On a couple different occasions I have run into this exception at start up:
>
> Exception (org.apache.cassandra.exceptions.InvalidRequestException) encountered during startup: Unknown type <user type>
> org.apache.cassandra.exceptions.InvalidRequestException: Unknown type <user type>
>         at org.apache.cassandra.cql3.CQL3Type$Raw$RawUT.prepare(CQL3Type.java:745)
>         at org.apache.cassandra.cql3.CQL3Type$Raw.prepareInternal(CQL3Type.java:533)
>         at org.apache.cassandra.schema.CQLTypeParser.parse(CQLTypeParser.java:53)
>         at org.apache.cassandra.schema.SchemaKeyspace.createColumnFromRow(SchemaKeyspace.java:1052)
>         at org.apache.cassandra.schema.SchemaKeyspace.lambda$fetchColumns$12(SchemaKeyspace.java:1038)
>
> This was with Cassandra 3.0.12 running in Kubernetes, which means that IP address changes for the Cassandra node can and will happen. Nowhere in client code does the UDT get dropped. I came across https://issues.apache.org/jira/browse/CASSANDRA-13739 which got me wondering if this particular Cassandra node wound up with another version of the system_schema.types table which did not have the UDT.
>
> In what circumstances could I end up with multiple table directories for the tables in system_schema? Right now I am just guessing that I wound up with a newer (or different) version of the system_schema.types table. Unfortunately, I no longer have access to the environment to confirm/deny what was happening. I just want to better understand so I can avoid it in the future.
>
>
> - John