You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Senthilvel Rangaswamy (JIRA)" <ji...@apache.org> on 2012/07/13 21:48:34 UTC

[jira] [Created] (CASSANDRA-4438) System reboot

Senthilvel Rangaswamy created CASSANDRA-4438:
------------------------------------------------

             Summary: System reboot
                 Key: CASSANDRA-4438
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4438
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 1.1.2
         Environment: EC2, Amazon Linux, Ephemeral Store
            Reporter: Senthilvel Rangaswamy


Since we deployed 1.1.2 we have been noticing random nodes in the ring
reboots often. There is no real info in any logs. It is happening on different
ec2 instances. So we can't say it is an instance problem. Any ideas on how to
find out what's causing this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4438) System reboot

Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414007#comment-13414007 ] 

Brandon Williams commented on CASSANDRA-4438:
---------------------------------------------

As noted, this belongs on the ML, not jira.

That said, since Amazon Linux is being used, I'll point you to CASSANDRA-4225
                
> System reboot
> -------------
>
>                 Key: CASSANDRA-4438
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4438
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.2
>         Environment: EC2, Amazon Linux, Ephemeral Store
>            Reporter: Senthilvel Rangaswamy
>
> Since we deployed 1.1.2 we have been noticing random nodes in the ring
> reboots often. There is no real info in any logs. It is happening on different
> ec2 instances. So we can't say it is an instance problem. Any ideas on how to
> find out what's causing this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Comment Edited] (CASSANDRA-4438) System reboot

Posted by "Rik Schneider (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414005#comment-13414005 ] 

Rik Schneider edited comment on CASSANDRA-4438 at 7/13/12 8:08 PM:
-------------------------------------------------------------------

It appears to be load related and is happening on multiple clusters. Once a node reboots once, it seems to go into a death spiral of bootstrapping causing a high load and the host reboots soon reboots again.

We are using the datastax RPM packages with EC2 multi region snitch on 8 nodes in each of 2 regions.


                
      was (Author: dawookie):
    It appears to be load related. Once a node reboots once, it seems to go into a death spiral of bootstrapping causing a high load and the host reboots soon reboots again.

We are using the datastax RPM packages with EC2 multi region snitch on 8 nodes in each of 2 regions.


                  
> System reboot
> -------------
>
>                 Key: CASSANDRA-4438
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4438
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.2
>         Environment: EC2, Amazon Linux, Ephemeral Store
>            Reporter: Senthilvel Rangaswamy
>
> Since we deployed 1.1.2 we have been noticing random nodes in the ring
> reboots often. There is no real info in any logs. It is happening on different
> ec2 instances. So we can't say it is an instance problem. Any ideas on how to
> find out what's causing this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Resolved] (CASSANDRA-4438) System reboot

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis resolved CASSANDRA-4438.
---------------------------------------

    Resolution: Invalid

The user mailing list is a more appropriate troubleshooting venue.
                
> System reboot
> -------------
>
>                 Key: CASSANDRA-4438
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4438
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.2
>         Environment: EC2, Amazon Linux, Ephemeral Store
>            Reporter: Senthilvel Rangaswamy
>
> Since we deployed 1.1.2 we have been noticing random nodes in the ring
> reboots often. There is no real info in any logs. It is happening on different
> ec2 instances. So we can't say it is an instance problem. Any ideas on how to
> find out what's causing this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4438) System reboot

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414006#comment-13414006 ] 

Jonathan Ellis commented on CASSANDRA-4438:
-------------------------------------------

Why would a node already part of the ring re-bootstrap?
                
> System reboot
> -------------
>
>                 Key: CASSANDRA-4438
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4438
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.2
>         Environment: EC2, Amazon Linux, Ephemeral Store
>            Reporter: Senthilvel Rangaswamy
>
> Since we deployed 1.1.2 we have been noticing random nodes in the ring
> reboots often. There is no real info in any logs. It is happening on different
> ec2 instances. So we can't say it is an instance problem. Any ideas on how to
> find out what's causing this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4438) System reboot

Posted by "Rik Schneider (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414005#comment-13414005 ] 

Rik Schneider commented on CASSANDRA-4438:
------------------------------------------

It appears to be load related. Once a node reboots once, it seems to go into a death spiral of bootstrapping causing a high load and the host reboots soon reboots again.

We are using the datastax RPM packages with EC2 multi region snitch on 8 nodes in each of 2 regions.


                
> System reboot
> -------------
>
>                 Key: CASSANDRA-4438
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4438
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.2
>         Environment: EC2, Amazon Linux, Ephemeral Store
>            Reporter: Senthilvel Rangaswamy
>
> Since we deployed 1.1.2 we have been noticing random nodes in the ring
> reboots often. There is no real info in any logs. It is happening on different
> ec2 instances. So we can't say it is an instance problem. Any ideas on how to
> find out what's causing this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira