You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Mina Naguib (JIRA)" <ji...@apache.org> on 2012/10/30 21:20:12 UTC
[jira] [Created] (CASSANDRA-4880) Endless loop flushing+compacting
system/schema_keyspaces and system/schema_columnfamilies
Mina Naguib created CASSANDRA-4880:
--------------------------------------
Summary: Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
Key: CASSANDRA-4880
URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.6
Environment: Linux x86_64 3.4.9
Reporter: Mina Naguib
After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
Stopping and starting the node entered the same loop.
Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4880) Endless loop flushing+compacting
system/schema_keyspaces and system/schema_columnfamilies
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reassigned CASSANDRA-4880:
-------------------------------------------
Assignee: Pavel Yaskevich
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493525#comment-13493525 ]
Pavel Yaskevich commented on CASSANDRA-4880:
--------------------------------------------
[~brandon.williams] Can you attach schema_* sstables that could trigger this behavior to the ticket please (if you have them)?
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502240#comment-13502240 ]
Brandon Williams commented on CASSANDRA-4880:
---------------------------------------------
+1
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6, 1.2.0 beta 1
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
> Fix For: 1.1.7, 1.2.0 rc1
>
> Attachments: CASSANDRA-4880-fix.patch, CASSANDRA-4880.patch
>
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493489#comment-13493489 ]
Brandon Williams commented on CASSANDRA-4880:
---------------------------------------------
Bisects to CASSANDRA-4561
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493529#comment-13493529 ]
Brandon Williams commented on CASSANDRA-4880:
---------------------------------------------
Sent to Pavel privately.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4880) Endless loop flushing+compacting
system/schema_keyspaces and system/schema_columnfamilies
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-4880:
----------------------------------------
Fix Version/s: 1.2.0 rc1
1.1.7
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6, 1.2.0 beta 1
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
> Fix For: 1.1.7, 1.2.0 rc1
>
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4880) Endless loop flushing+compacting
system/schema_keyspaces and system/schema_columnfamilies
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4880:
---------------------------------------
Attachment: CASSANDRA-4880-fix.patch
We don't need special RM deserialization handler any more, verified that tests are now passing.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6, 1.2.0 beta 1
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
> Fix For: 1.1.7, 1.2.0 rc1
>
> Attachments: CASSANDRA-4880-fix.patch, CASSANDRA-4880.patch
>
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493676#comment-13493676 ]
Pavel Yaskevich commented on CASSANDRA-4880:
--------------------------------------------
[~mheffner] and [~minaguib] Can you please execute following commands in CLI and post output here - "use system;" and "list schema_keyspaces"?
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4880) Endless loop flushing+compacting
system/schema_keyspaces and system/schema_columnfamilies
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4880:
---------------------------------------
Attachment: CASSANDRA-4880.patch
I made start timestamp schema timestamp repair account and fix any timestamps that are grater than current timestamp and introduced new version in MessagingVersion as it's the only way to fix infinite loop in schema due to broken nano timestamps that we can't properly fix in <= 1.1.6.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6, 1.2.0 beta 1
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
> Fix For: 1.1.7, 1.2.0 rc1
>
> Attachments: CASSANDRA-4880.patch
>
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497524#comment-13497524 ]
Pavel Yaskevich edited comment on CASSANDRA-4880 at 11/14/12 10:16 PM:
-----------------------------------------------------------------------
I made start timestamp schema timestamp repair account and fix any timestamps that are grater than current timestamp and introduced new version in MessagingVersion as it's the only way to fix infinite loop in schema due to broken nano timestamps that we can't properly fix in <= 1.1.6.
Edit: Also note that once this is committed we would have to increase VERSION number for cassandra-1.2.
was (Author: xedin):
I made start timestamp schema timestamp repair account and fix any timestamps that are grater than current timestamp and introduced new version in MessagingVersion as it's the only way to fix infinite loop in schema due to broken nano timestamps that we can't properly fix in <= 1.1.6.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6, 1.2.0 beta 1
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
> Fix For: 1.1.7, 1.2.0 rc1
>
> Attachments: CASSANDRA-4880.patch
>
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4880) Endless loop flushing+compacting
system/schema_keyspaces and system/schema_columnfamilies
Posted by "Mina Naguib (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mina Naguib updated CASSANDRA-4880:
-----------------------------------
Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33 (was: Linux x86_64 3.4.9)
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498151#comment-13498151 ]
Brandon Williams commented on CASSANDRA-4880:
---------------------------------------------
Hmm, this unfortunate but I don't see any way around it, either. +1
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6, 1.2.0 beta 1
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
> Fix For: 1.1.7, 1.2.0 rc1
>
> Attachments: CASSANDRA-4880.patch
>
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493737#comment-13493737 ]
Pavel Yaskevich commented on CASSANDRA-4880:
--------------------------------------------
it looks like it wasn't a good idea to involve timestamps in computing schema version (was super easy to do), I will work on improving things to make a migration from older versions (which System.nanoTime() bug) as smooth as possible.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493493#comment-13493493 ]
Pavel Yaskevich commented on CASSANDRA-4880:
--------------------------------------------
can you try 1.1.6 + CASSANDRA-4837 ?
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501508#comment-13501508 ]
Brandon Williams commented on CASSANDRA-4880:
---------------------------------------------
Nope, those tests all run with the same version.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6, 1.2.0 beta 1
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
> Fix For: 1.1.7, 1.2.0 rc1
>
> Attachments: CASSANDRA-4880.patch
>
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501504#comment-13501504 ]
Pavel Yaskevich commented on CASSANDRA-4880:
--------------------------------------------
Could that be that those two stub nodes running on different versions - one on recent and one on previous? That would explain why they have disagreement. It's tested on rolling restart by CASSANDRA-4837.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6, 1.2.0 beta 1
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
> Fix For: 1.1.7, 1.2.0 rc1
>
> Attachments: CASSANDRA-4880.patch
>
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493522#comment-13493522 ]
Brandon Williams commented on CASSANDRA-4880:
---------------------------------------------
No help :(
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493414#comment-13493414 ]
Brandon Williams commented on CASSANDRA-4880:
---------------------------------------------
I'm pretty sure that's what's happening, but it's not clear why. You can artificially reproduce this by manually injecting a conflict, such as a "Column family ID mismatch" though that error isn't present here.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Mike Heffner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493355#comment-13493355 ]
Mike Heffner commented on CASSANDRA-4880:
-----------------------------------------
We also see this after upgrading a node from 1.1.0 -> 1.1.6 in our ring. We also saw increased flush writes begin continuously on the other 1.1.0 nodes in the ring after the 1.1.6 was started, but not as rapidly as on the 1.1.6 node. There were also periodic HintedHandoffs occurring around the ring after the 1.1.6 node came up.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501511#comment-13501511 ]
Pavel Yaskevich commented on CASSANDRA-4880:
--------------------------------------------
Interesting... Ok, I will try to figure it out asap.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6, 1.2.0 beta 1
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
> Fix For: 1.1.7, 1.2.0 rc1
>
> Attachments: CASSANDRA-4880.patch
>
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493372#comment-13493372 ]
Jonathan Ellis commented on CASSANDRA-4880:
-------------------------------------------
Is schema synchronization confused? I can't think of any other reason for continuous flush here.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4880) Endless loop flushing+compacting
system/schema_keyspaces and system/schema_columnfamilies
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-4880:
----------------------------------------
Affects Version/s: 1.2.0 beta 1
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6, 1.2.0 beta 1
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
> Fix For: 1.1.7, 1.2.0 rc1
>
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Mike Heffner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493702#comment-13493702 ]
Mike Heffner commented on CASSANDRA-4880:
-----------------------------------------
This is after we removed the 1.1.6 node, but:
{code}
Using default limit of 100
Using default column limit of 100
-------------------
RowKey: KSName
=> (column=durable_writes, value=true, timestamp=5397847727343)
=> (column=name, value=KSName, timestamp=5397847727343)
=> (column=strategy_class, value=org.apache.cassandra.locator.NetworkTopologyStrategy, timestamp=5397847727343)
=> (column=strategy_options, value={"us-east":"3"}, timestamp=5397847727343)
1 Row Returned.
Elapsed time: 132 msec(s).
{code}
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4880) Endless loop
flushing+compacting system/schema_keyspaces and
system/schema_columnfamilies
Posted by "Mina Naguib (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493692#comment-13493692 ]
Mina Naguib commented on CASSANDRA-4880:
----------------------------------------
[~xedin] Here's mine (after some very minor name obfuscation):
{code}
[default@system] list schema_keyspaces;
Using default limit of 100
Using default column limit of 100
-------------------
RowKey: AdKS
=> (column=durable_writes, value=true, timestamp=1336233003898)
=> (column=name, value=AdKS, timestamp=1336233003898)
=> (column=strategy_class, value=org.apache.cassandra.locator.NetworkTopologyStrategy, timestamp=1336233003898)
=> (column=strategy_options, value={"DCMTL":"2","DCLA":"2","DCLON":"2"}, timestamp=1336233003898)
1 Row Returned.
Elapsed time: 3 msec(s).
{code}
Tomorrow I'll run a test with cassandra 1.1.6 + CASSANDRA-4837 just to add a second vote to Brandon's test whether it helps or not.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-4880) Endless loop flushing+compacting
system/schema_keyspaces and system/schema_columnfamilies
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reopened CASSANDRA-4880:
-----------------------------------------
I think something's still wrong in 1.1, concurrent schema changes aren't settling: http://buildbot.datastax.com:8010/builders/cassandra-1.1/builds/567/steps/shell/logs/stdio
Everything works in 1.2.
> Endless loop flushing+compacting system/schema_keyspaces and system/schema_columnfamilies
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4880
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6, 1.2.0 beta 1
> Environment: Linux x86_64 3.4.9, sun-jdk 1.6.0_33
> Reporter: Mina Naguib
> Assignee: Pavel Yaskevich
> Fix For: 1.1.7, 1.2.0 rc1
>
> Attachments: CASSANDRA-4880.patch
>
>
> After upgrading a node from 1.1.2 to 1.1.6, the startup sequence entered a loop as seen here:
> http://mina.naguib.ca/misc/cassandra_116_startup_loop.txt
> Stopping and starting the node entered the same loop.
> Reverting back to 1.1.2 started successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira