You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "David Daeschler (Created) (JIRA)" <ji...@apache.org> on 2012/03/20 16:45:39 UTC
[jira] [Created] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
-----------------------------------------------------------------------------------------
Key: CASSANDRA-4066
URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.0.6
Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
Reporter: David Daeschler
The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Posted by "Herman Meerlo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404712#comment-13404712 ]
Herman Meerlo commented on CASSANDRA-4066:
------------------------------------------
Same here, certainly no kernel panics but nodes in the cluster just started marking each other as being dead:
INFO [GossipTasks:1] 2012-07-01 02:00:55,110 Gossiper.java (line 818) InetAddress /192.168.20.139 is now dead.
INFO [GossipStage:1] 2012-07-01 02:02:41,853 Gossiper.java (line 804) InetAddress /192.168.20.139 is now UP
> Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
> Reporter: David Daeschler
> Assignee: Brandon Williams
> Priority: Minor
> Labels: gossip
> Fix For: 1.1.1
>
>
> The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
> Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
> I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
> Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404692#comment-13404692 ]
Jonathan Ellis commented on CASSANDRA-4066:
-------------------------------------------
http://serverfault.com/a/403767
> Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
> Reporter: David Daeschler
> Assignee: Brandon Williams
> Priority: Minor
> Labels: gossip
> Fix For: 1.1.1
>
>
> The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
> Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
> I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
> Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Posted by "David Daeschler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233521#comment-13233521 ]
David Daeschler commented on CASSANDRA-4066:
--------------------------------------------
Seeing the reference bug you posted, this might help as well:
# java -version
java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
> Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
> Reporter: David Daeschler
> Assignee: Brandon Williams
> Priority: Minor
> Labels: gossip
> Fix For: 1.1.1
>
>
> The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
> Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
> I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
> Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405110#comment-13405110 ]
Jonathan Ellis commented on CASSANDRA-4066:
-------------------------------------------
To summarize:
The linux kernel bug causes livelock when ntp moves the clock backwards. This manifests with nodes showing super high load for no [other] reason and marking each other down as Herman points out, both on Java systems like C*/Hadoop/ElasticSearch and non-Java systems like mysql. It does NOT cause the kind of GossipTask death Brandon talks about in the first part of the ticket, which you only need to worry about if you're changing time by a lot more than a leap second.
The former you can fix by upgrading to a fixed linux kernel before the next leap second in a few years, or by rebooting or applying {{date; date `date +"%m%d%H%M%C%y.%S"`; date}} when you get bitten. The latter is not worth addressing since it only applies to VMs that get hibernated.
> Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
> Reporter: David Daeschler
> Assignee: Brandon Williams
> Priority: Minor
> Labels: gossip
> Fix For: 1.1.1
>
>
> The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
> Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
> I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
> Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Posted by "David Daeschler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233526#comment-13233526 ]
David Daeschler commented on CASSANDRA-4066:
--------------------------------------------
Another bit of information. I tried the normal procedure for stopping the server which is part of an init script. It executes 'kill PID' to stop the server. This script failed after waiting about 5 minutes for cassandra to stop. Checking top, I didn't see any CPU or disk activity from the cassandra service, so I ended up force killing it with kill -9.
Just giving this information in case it affects your view of the scope/location of the problem.
> Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
> Reporter: David Daeschler
> Assignee: Brandon Williams
> Priority: Minor
> Labels: gossip
> Fix For: 1.1.1
>
>
> The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
> Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
> I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
> Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4066:
--------------------------------------
Component/s: Core
Priority: Minor (was: Major)
Affects Version/s: (was: 1.0.6)
Fix Version/s: 1.1.1
Assignee: Brandon Williams
Labels: gossip (was: )
We make extensive use of Java's ScheduledExecutorService, which does not deal well with the system time being pulled out from under it: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7139684
I'm willing to live with this for the majority of scheduled tasks, however, it might be worth updating Gossip to use it's own thread + sleep calls to avoid this.
On the other hand, if you didn't have Gossip dying with UAE, it would be very difficult to figure out why the rest of the background tasks stopped executing, which would cause things to go bad a lot more gradually.
What do you think, Brandon?
> Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
> Reporter: David Daeschler
> Assignee: Brandon Williams
> Priority: Minor
> Labels: gossip
> Fix For: 1.1.1
>
>
> The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
> Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
> I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
> Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233648#comment-13233648 ]
Brandon Williams commented on CASSANDRA-4066:
---------------------------------------------
I can confirm that this is indeed the GossipTask no longer running when the clock is pushed backward far enough. As you mention, we use SES extensively and likely all those timed tasks have also quit firing, which could lead to an untold amount of confusion if we special-cased gossip, since there would be no immediate red flag to indicate a problem. Starting a node far in the future has other consequences too, such as CASSANDRA-3654. I think I would rather see the UAE and know that my machines have connectivity to identify this problem and fix it correctly.
Even if we do special-case the GossipTask, we'll also need to fix LoadBroadcaster so we don't end up with a broken view of the load on the ring, and at that point it feels like a slippery slope where we need to fix everything, or fail as quickly as possible, which is what the current behavior does.
Also an interesting thing to note is that the node still replies to gossip syn messages with a gossip ack, but because we only update the FD on a version/generation change, and because LoadBroadcaster is also broken the node has no reason to generate new versions, it remains seen as down to the other nodes. If LB did happen to work, we'd see the node flap every 90 seconds.
> Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
> Reporter: David Daeschler
> Assignee: Brandon Williams
> Priority: Minor
> Labels: gossip
> Fix For: 1.1.1
>
>
> The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
> Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
> I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
> Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404685#comment-13404685 ]
Jonathan Ellis commented on CASSANDRA-4066:
-------------------------------------------
It wouldn't have; those who were bit by the linux kernel bug had worse problems than this to worry about.
> Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
> Reporter: David Daeschler
> Assignee: Brandon Williams
> Priority: Minor
> Labels: gossip
> Fix For: 1.1.1
>
>
> The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
> Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
> I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
> Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Posted by "David Daeschler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233662#comment-13233662 ]
David Daeschler commented on CASSANDRA-4066:
--------------------------------------------
Thanks for looking. At least this is documented now.
Just FYI this can become a problem on some virtualized systems even with time sync enabled. For example there appears to be a bug going around for windows on Hyper-V where the time sync service just stops working. I have a lot of servers running in this configuration (though not running cassandra thankfully) and I have had to schedule a daily reset of the time sync services to keep the clocks in check.
Anyways, have a nice day and keep up the good work.
> Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
> Reporter: David Daeschler
> Assignee: Brandon Williams
> Priority: Minor
> Labels: gossip
> Fix For: 1.1.1
>
>
> The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
> Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
> I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
> Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Posted by "David Daeschler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404680#comment-13404680 ]
David Daeschler commented on CASSANDRA-4066:
--------------------------------------------
I just wanted to bump this bug in case it would've helped with the leap second issue that affected many businesses tonight
> Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
> Reporter: David Daeschler
> Assignee: Brandon Williams
> Priority: Minor
> Labels: gossip
> Fix For: 1.1.1
>
>
> The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
> Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
> I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
> Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Posted by "Brandon Williams (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams resolved CASSANDRA-4066.
-----------------------------------------
Resolution: Won't Fix
> Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
> Reporter: David Daeschler
> Assignee: Brandon Williams
> Priority: Minor
> Labels: gossip
> Fix For: 1.1.1
>
>
> The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
> Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
> I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
> Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4066) Cassandra cluster stops
responding on time change (scheduling not using monotonic time?)
Posted by "David Daeschler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404690#comment-13404690 ]
David Daeschler commented on CASSANDRA-4066:
--------------------------------------------
Hi Johnathan,
I don't have a lot of detailed knowledge about the linux/java/leap second issue, but in my case there was no kernel panic or any other kernel issues. No other software on the exact same OS was affected. Just Java/cassandra instances spinning cores wildly. This seems to be the case with a few others I've spoken to as well.
> Cassandra cluster stops responding on time change (scheduling not using monotonic time?)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4066
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux; CentOS6 2.6.32-220.4.2.el6.x86_64
> Reporter: David Daeschler
> Assignee: Brandon Williams
> Priority: Minor
> Labels: gossip
> Fix For: 1.1.1
>
>
> The server installation I set up did not have ntpd installed in the base installation. When I noticed that the clocks were skewing I installed ntp and set the date on all the servers in the cluster. A short time later, I started getting UnavailableExceptions on the clients.
> Also, one sever seemed to be unaffected by the time change. That server happened to have it's time pushed forward, not backwards like the other 3 in the cluster. This leads me to believe something is running on a timer/schedule that is not monotonic.
> I'm posting this as a bug, but I suppose it might just be part of the communication protocols etc for the cluster and part of the design. But I think the devs should be aware of what I saw.
> Otherwise, thank you for a fantastic product. Even after restarting 75% of the cluster things seem to have recovered nicely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira