You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "Tom White (Created) (JIRA)" <ji...@apache.org> on 2012/03/09 07:37:57 UTC

[jira] [Created] (WHIRR-552) Fix sporadic HBase 0.92 failures

Fix sporadic HBase 0.92 failures
--------------------------------

                 Key: WHIRR-552
                 URL: https://issues.apache.org/jira/browse/WHIRR-552
             Project: Whirr
          Issue Type: Bug
          Components: service/hbase
            Reporter: Tom White
             Fix For: 0.7.2


Occasionally the HBase 0.92 service fails to start. See WHIRR-525 for a description.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (WHIRR-552) Fix sporadic HBase 0.92 failures

Posted by "Tom White (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/WHIRR-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated WHIRR-552:
----------------------------

    Fix Version/s:     (was: 0.8.0)
                   0.8.1
    
> Fix sporadic HBase 0.92 failures
> --------------------------------
>
>                 Key: WHIRR-552
>                 URL: https://issues.apache.org/jira/browse/WHIRR-552
>             Project: Whirr
>          Issue Type: Bug
>          Components: service/hbase
>            Reporter: Tom White
>             Fix For: 0.8.1
>
>         Attachments: master.log, rs.log, zk.log
>
>
> Occasionally the HBase 0.92 service fails to start. See WHIRR-525 for a description.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (WHIRR-552) Fix sporadic HBase 0.92 failures

Posted by "Andrew Bayer (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/WHIRR-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Bayer updated WHIRR-552:
-------------------------------

    Attachment: WHIRR-552.patch

Switching to HBase 0.92.2 seems to do the trick.
                
> Fix sporadic HBase 0.92 failures
> --------------------------------
>
>                 Key: WHIRR-552
>                 URL: https://issues.apache.org/jira/browse/WHIRR-552
>             Project: Whirr
>          Issue Type: Bug
>          Components: service/hbase
>            Reporter: Tom White
>            Assignee: Andrew Bayer
>             Fix For: 0.9.0
>
>         Attachments: master.log, rs.log, WHIRR-552.patch, zk.log
>
>
> Occasionally the HBase 0.92 service fails to start. See WHIRR-525 for a description.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (WHIRR-552) Fix sporadic HBase 0.92 failures

Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226333#comment-13226333 ] 

Tom White commented on WHIRR-552:
---------------------------------

Patrick Hunt told me offline that the "Connected to an old server; r-o mode will be unavailable" message is not a problem. Also, worth trying telnet not nc, since nc sometimes has issues.

I wonder if this is a timing issue, and simply making the RS wait a bit would help.
                
> Fix sporadic HBase 0.92 failures
> --------------------------------
>
>                 Key: WHIRR-552
>                 URL: https://issues.apache.org/jira/browse/WHIRR-552
>             Project: Whirr
>          Issue Type: Bug
>          Components: service/hbase
>            Reporter: Tom White
>             Fix For: 0.7.2
>
>         Attachments: master.log, rs.log, zk.log
>
>
> Occasionally the HBase 0.92 service fails to start. See WHIRR-525 for a description.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (WHIRR-552) Fix sporadic HBase 0.92 failures

Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/WHIRR-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated WHIRR-552:
----------------------------

    Attachment: zk.log
                master.log
                rs.log

Here are the log files from a time that the service failed.
                
> Fix sporadic HBase 0.92 failures
> --------------------------------
>
>                 Key: WHIRR-552
>                 URL: https://issues.apache.org/jira/browse/WHIRR-552
>             Project: Whirr
>          Issue Type: Bug
>          Components: service/hbase
>            Reporter: Tom White
>             Fix For: 0.7.2
>
>         Attachments: master.log, rs.log, zk.log
>
>
> Occasionally the HBase 0.92 service fails to start. See WHIRR-525 for a description.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (WHIRR-552) Fix sporadic HBase 0.92 failures

Posted by "Karel Vervaeke (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242141#comment-13242141 ] 

Karel Vervaeke commented on WHIRR-552:
--------------------------------------

Here's what I got:

5:17:23 RS starting up, failure to connect to zk (logical since zk not started yet)
5:17:42 Master starting up, same kind of failures
5:17:47 ZK starting
5:17:47 RS connects to zk
5:17:47 Master connects to zk
5:17:53 Master: found 1 replicas but expecting no less than 3
5:17:54 - 5:18:02 Master 'waiting for rs to check in'
5:17:58 RS: first signs of stopping (aborting, initialization of fs failed, bla bla)

The highlights to me are
1) 5:17:53 Master:found 1 replicas but expecting no less than 3
Wrong value for dfs.replication? Which value does it have in the datanodes?

2) Why did the rs not check in between 5:17:48 and :58? No activity in the rs logs...

3) The master logs appear truncated. Is it just part of the log or does it really end suddenly?
Memory issues?
                
> Fix sporadic HBase 0.92 failures
> --------------------------------
>
>                 Key: WHIRR-552
>                 URL: https://issues.apache.org/jira/browse/WHIRR-552
>             Project: Whirr
>          Issue Type: Bug
>          Components: service/hbase
>            Reporter: Tom White
>             Fix For: 0.7.2
>
>         Attachments: master.log, rs.log, zk.log
>
>
> Occasionally the HBase 0.92 service fails to start. See WHIRR-525 for a description.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Assigned] (WHIRR-552) Fix sporadic HBase 0.92 failures

Posted by "Andrew Bayer (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/WHIRR-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Bayer reassigned WHIRR-552:
----------------------------------

    Assignee: Andrew Bayer
    
> Fix sporadic HBase 0.92 failures
> --------------------------------
>
>                 Key: WHIRR-552
>                 URL: https://issues.apache.org/jira/browse/WHIRR-552
>             Project: Whirr
>          Issue Type: Bug
>          Components: service/hbase
>            Reporter: Tom White
>            Assignee: Andrew Bayer
>             Fix For: 0.9.0
>
>         Attachments: master.log, rs.log, zk.log
>
>
> Occasionally the HBase 0.92 service fails to start. See WHIRR-525 for a description.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (WHIRR-552) Fix sporadic HBase 0.92 failures

Posted by "Andrei Savu (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/WHIRR-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrei Savu updated WHIRR-552:
------------------------------

    Fix Version/s:     (was: 0.7.2)
                   0.8.0
    
> Fix sporadic HBase 0.92 failures
> --------------------------------
>
>                 Key: WHIRR-552
>                 URL: https://issues.apache.org/jira/browse/WHIRR-552
>             Project: Whirr
>          Issue Type: Bug
>          Components: service/hbase
>            Reporter: Tom White
>             Fix For: 0.8.0
>
>         Attachments: master.log, rs.log, zk.log
>
>
> Occasionally the HBase 0.92 service fails to start. See WHIRR-525 for a description.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (WHIRR-552) Fix sporadic HBase 0.92 failures

Posted by "Amandeep Khurana (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242829#comment-13242829 ] 

Amandeep Khurana commented on WHIRR-552:
----------------------------------------

I ran into issues where the parent znode was not created because the master had not yet initialized. RS came up and died because of that.
Related issue - https://issues.apache.org/jira/browse/HBASE-5666.
                
> Fix sporadic HBase 0.92 failures
> --------------------------------
>
>                 Key: WHIRR-552
>                 URL: https://issues.apache.org/jira/browse/WHIRR-552
>             Project: Whirr
>          Issue Type: Bug
>          Components: service/hbase
>            Reporter: Tom White
>             Fix For: 0.8.0
>
>         Attachments: master.log, rs.log, zk.log
>
>
> Occasionally the HBase 0.92 service fails to start. See WHIRR-525 for a description.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira