You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Wade Simmons (JIRA)" <ji...@apache.org> on 2012/10/19 20:49:12 UTC
[jira] [Created] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Wade Simmons created CASSANDRA-4837:
---------------------------------------
Summary: IllegalStateException when upgrading schema
Key: CASSANDRA-4837
URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.6
Environment: Linux
Reporter: Wade Simmons
I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting the second node with new code, I am seeing this exception repeat in the logs:
{code}
ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
java.lang.IllegalStateException: One row required, 0 found
at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
I added in some debugging logging to see what Row it was trying to load, and I see this:
{code}
Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
{code}
The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich resolved CASSANDRA-4837.
----------------------------------------
Resolution: Duplicate
Resolving as a duplicate to CASSANDRA-4880.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485038#comment-13485038 ]
Wade Simmons commented on CASSANDRA-4837:
-----------------------------------------
{quote}
I have tried running latest build of cassandra-1.1 branch with your schema sstables and it worked for me just fine, I can see an "atlas" keyspace and all of it's CFs, can you try building and running latest cassandra-1.1 branch instead of 1.1.6 release on that node to verify?
{quote}
To properly test this you will need to start a small 1.1.2 cluster with this schema and then upgrade one node to 1.1.6. The exception only happens when getting schema gossip from old nodes (that think they have a newer schema than the 1.1.6 node).
I will try this patch.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483404#comment-13483404 ]
Wade Simmons commented on CASSANDRA-4837:
-----------------------------------------
AL: I haven't been able to get it to work yet. Still trying to find a solution that works.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "AL (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482505#comment-13482505 ]
AL commented on CASSANDRA-4837:
-------------------------------
Hi, we encountered the same problem as we upgrade from 1.1.1 -> 1.1.6.
I tried copied the schema_* data over from the 1-node cluster, same log msg pops up. :( Wondering if it works for you Wade?
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4837:
---------------------------------------
Attachment: CASSANDRA-4837.patch
patch removes deleted empty CFs from remote schema mutations. [~wadey] please try running latest cassandra-1.1 with this patch applied.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481641#comment-13481641 ]
Wade Simmons commented on CASSANDRA-4837:
-----------------------------------------
If I try ro recreate the schema with the CLI with gossip disabled (I used join_ring=false) I get exceptions because it gets NPEs trying to send the schema over gossip.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489634#comment-13489634 ]
Wade Simmons edited comment on CASSANDRA-4837 at 11/2/12 6:29 PM:
------------------------------------------------------------------
[~xedin]: When you tested, did you start a small cluster with 1.1.2 nodes and then upgrade one to 1.1.6?
was (Author: wadey):
Pavel: When you tested, did you start a small cluster with 1.1.2 nodes and then upgrade one to 1.1.6?
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wade Simmons updated CASSANDRA-4837:
------------------------------------
Attachment: schematables.tar.gz
schematables from 1.1.2 node. Trying to upgrade one node in the cluster to 1.1.6 causes migration errors from nodes that have these tables.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489651#comment-13489651 ]
Pavel Yaskevich commented on CASSANDRA-4837:
--------------------------------------------
[~wadey] Yes I did start 3 node cluster using ccm on my local machine, let me modify a patch a little bit so you can try running it again...
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484680#comment-13484680 ]
Pavel Yaskevich commented on CASSANDRA-4837:
--------------------------------------------
Actually let me add one change to the remote mutation deserialization to be sure that it correctly handles deleted data.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481785#comment-13481785 ]
Pavel Yaskevich commented on CASSANDRA-4837:
--------------------------------------------
Yes, this is what I wanted to suggest.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481681#comment-13481681 ]
Wade Simmons commented on CASSANDRA-4837:
-----------------------------------------
My next attempt will be to recreate the schema on a new 1-node cluster, and then copy the resulting sstables over.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499195#comment-13499195 ]
Pavel Yaskevich commented on CASSANDRA-4837:
--------------------------------------------
[~wadey] Can you please check if patch from CASSANDRA-4880 helps you?
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4837:
---------------------------------------
Attachment: (was: CASSANDRA-4837.patch)
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4837:
---------------------------------------
Attachment: (was: CASSANDRA-4837.patch)
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487426#comment-13487426 ]
Wade Simmons commented on CASSANDRA-4837:
-----------------------------------------
I see the same exception loop when applying the patch to 1.1.6.
What would be different in cassandra-1.1 that could possibly fix it? (this is a staging cluster, so I would rather not upgrade to an unreleased version if it is just a guess). I don't see anything that looks related to schema in the diff: https://github.com/apache/cassandra/compare/cassandra-1.1.6...cassandra-1.1
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480596#comment-13480596 ]
Pavel Yaskevich commented on CASSANDRA-4837:
--------------------------------------------
Try following - stop that node, delete all files in schema_*, start the node with disabled gossip, re-create schema using CLI and reintroduce node to the ring.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484293#comment-13484293 ]
Wade Simmons commented on CASSANDRA-4837:
-----------------------------------------
Perhaps the fix is to make 1.1.6 nodes ignore gossip schema from 1.1.2 nodes?
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489634#comment-13489634 ]
Wade Simmons commented on CASSANDRA-4837:
-----------------------------------------
Pavel: When you tested, did you start a small cluster with 1.1.2 nodes and then upgrade one to 1.1.6?
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484676#comment-13484676 ]
Pavel Yaskevich commented on CASSANDRA-4837:
--------------------------------------------
I have tried running latest build of cassandra-1.1 branch with your schema sstables and it worked for me just fine, I can see an "atlas" keyspace and all of it's CFs, can you try building and running latest cassandra-1.1 branch instead of 1.1.6 release on that node to verify?
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480249#comment-13480249 ]
Wade Simmons commented on CASSANDRA-4837:
-----------------------------------------
The fact that the CF for schema_keyspaces is marked for delete worries me, but perhaps that is an artifacts of the schema merge?
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting the second node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4837:
---------------------------------------
Attachment: CASSANDRA-4837.patch
updated patch rejects parts of the remote schema which represent empty expired rows.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4837:
---------------------------------------
Attachment: CASSANDRA-4837.patch
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483436#comment-13483436 ]
Pavel Yaskevich commented on CASSANDRA-4837:
--------------------------------------------
[~wadey] Can you please attach schema_* directories here so I can take a look what is going on there?
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483554#comment-13483554 ]
Wade Simmons edited comment on CASSANDRA-4837 at 10/24/12 10:04 PM:
--------------------------------------------------------------------
schematables from 1.1.2 node. Trying to upgrade one node in the cluster to 1.1.6 causes migration errors from 1.1.2 nodes that have these tables.
You will notice that we have a modification in our system to allow "@" chars in CF names, but that shouldn't be causing the issue here.
was (Author: wadey):
schematables from 1.1.2 node. Trying to upgrade one node in the cluster to 1.1.6 causes migration errors from nodes that have these tables.
You will notice that we have a modification in our system to allow "@" chars in CF names, but that shouldn't be causing the issue here.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499248#comment-13499248 ]
Wade Simmons commented on CASSANDRA-4837:
-----------------------------------------
I applied just the patch for CASSANDRA-4880 to 1.1.6 (removed the previous patch here) and it appears to have fixed the issue. As I upgrade nodes, their schema version is different (but equal to each other) and they are ignoring the schema from the old nodes.
I recommend closing this issue as a duplicate of CASSANDRA-4880.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wade Simmons updated CASSANDRA-4837:
------------------------------------
Description:
I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
{code}
ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
java.lang.IllegalStateException: One row required, 0 found
at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
I added in some debugging logging to see what Row it was trying to load, and I see this:
{code}
Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
{code}
The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
was:
I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting the second node with new code, I am seeing this exception repeat in the logs:
{code}
ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
java.lang.IllegalStateException: One row required, 0 found
at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
I added in some debugging logging to see what Row it was trying to load, and I see this:
{code}
Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
{code}
The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496601#comment-13496601 ]
Wade Simmons commented on CASSANDRA-4837:
-----------------------------------------
Same exception repeating. I'm going to see if I can make a patch to avoid this.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480464#comment-13480464 ]
Wade Simmons commented on CASSANDRA-4837:
-----------------------------------------
I removed system/schema_* and commitlog/* from the troublesome node, restarted it and it ran into the same problem (it grabbed the schema from gossip). Do I need to disable gossip on the node somehow so it doesn't run into this issue? Is there any way to fix this without having to stop the whole cluster?
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting the second node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480398#comment-13480398 ]
Pavel Yaskevich commented on CASSANDRA-4837:
--------------------------------------------
[~wadey] It looks like related to CASSANDRA-4698. Can you dump the user-defined schema using CLI 'show schema' command, remove all sstables from system/schema_* directories on the nodes and try to re-create schema back using CLI, that would fix the problem with deleted-at and you will be able to do a safe rolling restart.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting the second node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483554#comment-13483554 ]
Wade Simmons edited comment on CASSANDRA-4837 at 10/24/12 8:24 PM:
-------------------------------------------------------------------
schematables from 1.1.2 node. Trying to upgrade one node in the cluster to 1.1.6 causes migration errors from nodes that have these tables.
You will notice that we have a modification in our system to allow "@" chars in CF names, but that shouldn't be causing the issue here.
was (Author: wadey):
schematables from 1.1.2 node. Trying to upgrade one node in the cluster to 1.1.6 causes migration errors from nodes that have these tables.
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Wade Simmons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480396#comment-13480396 ]
Wade Simmons commented on CASSANDRA-4837:
-----------------------------------------
It looks like the issue might be that we have had two different keyspaces with different capitalization in the past. I logged the resulting MapDifference generated in mergeKeyspaces, here it is (with some whitespace added):
{code}
not equal: value differences={
DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374)=(
ColumnFamily(schema_keyspaces -deleted at 1350680301441000- []),
ColumnFamily(schema_keyspaces -deleted at 1350680302220000- [])
),
DecoratedKey(100476189400466680783670335581709524812, 7365676d656e7473696e746567726174696f6e74657374)=(
ColumnFamily(schema_keyspaces -deleted at 1350680301441000- []),
ColumnFamily(schema_keyspaces -deleted at 1350680302220000- [])
),
DecoratedKey(5845054961105273922406180493871966218, 5365676d656e7473)=(
ColumnFamily(schema_keyspaces [durable_writes:false:1@1350680301442000,name:false:8@1350680301442000,strategy_class:false:43@1350680301442000,strategy_options:false:26@1350680301442000,]),
ColumnFamily(schema_keyspaces [durable_writes:false:1@1350680302220000,name:false:8@1350680302220000,strategy_class:false:43@1350680302220000,strategy_options:false:26@1350680302220000,])
),
}
{code}
The two keyspaces with the CF marked for deletion are called "SegmentsIntegrationTest" and "segmentsintegrationtest".
If I do a "select * from system.schema_keyspaces;" on a node that is still on the old version, I only see the keyspace "SegmentsIntegrationTest".
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting the second node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reassigned CASSANDRA-4837:
-------------------------------------------
Assignee: Pavel Yaskevich
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting the second node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4837) IllegalStateException when
upgrading schema
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499255#comment-13499255 ]
Pavel Yaskevich commented on CASSANDRA-4837:
--------------------------------------------
Great! They would all agree on schema once you have finished upgrade. I will resolve as Duplicate, thanks!
> IllegalStateException when upgrading schema
> -------------------------------------------
>
> Key: CASSANDRA-4837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4837
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux
> Reporter: Wade Simmons
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4837.patch, schematables.tar.gz
>
>
> I am upgrading a cluster from 1.1.2 to 1.1.6. When restarting a node with new code, I am seeing this exception repeat in the logs:
> {code}
> ERROR [InternalResponseStage:21] 2012-10-19 00:41:26,794 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:21,5,main]
> java.lang.IllegalStateException: One row required, 0 found
> at org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:50)
> at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:258)
> at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:406)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:355)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:329)
> at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:449)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> I added in some debugging logging to see what Row it was trying to load, and I see this:
> {code}
> Unable to load keyspace schema: Row(key=DecoratedKey(112573196966143652100562749464385838776, 5365676d656e7473496e746567726174696f6e54657374), cf=ColumnFamily(schema_keyspaces -deleted at 1350665377628000- []))
> {code}
> The hex key translates to a schema that exists in schema_keyspaces when I query on the rest of the cluster. I tried restarting one of the other nodes without upgrading the jar and it restarted without exceptions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira