You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Eric Falcao (JIRA)" <ji...@apache.org> on 2011/08/13 00:49:27 UTC

[jira] [Created] (CASSANDRA-3023) NPE in describe_ring

NPE in describe_ring
--------------------

                 Key: CASSANDRA-3023
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.8.4
            Reporter: Eric Falcao


Not sure how much of the following is relevant besides the stack trace, but here I go:

I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:

-edited seeds
-stopped both DC1 nodes
-upgraded jars
-started both nodes at the same time

The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:

ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
java.lang.NullPointerException
	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3023) NPE in describe_ring

Posted by "Eric Falcao (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090489#comment-13090489 ] 

Eric Falcao commented on CASSANDRA-3023:
----------------------------------------

I think my clients may have rapidly been calling describe_ring.........hrm......which should have stopped NPE'ing after the full (but rolling) upgrade.

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>             Fix For: 0.8.5
>
>         Attachments: 3023.txt
>
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3023) NPE in describe_ring

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089944#comment-13089944 ] 

Jonathan Ellis commented on CASSANDRA-3023:
-------------------------------------------

I think "return half-broken data" is the right solution, since pre-1777 clients aren't going to be looking for the new data anyway.

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>             Fix For: 0.8.5
>
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Assigned] (CASSANDRA-3023) NPE in describe_ring

Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brandon Williams reassigned CASSANDRA-3023:
-------------------------------------------

    Assignee: Brandon Williams

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3023) NPE in describe_ring

Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090304#comment-13090304 ] 

Brandon Williams commented on CASSANDRA-3023:
---------------------------------------------

bq. Will problem disappear after full upgrade to 0.8.4?

Yes.

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>             Fix For: 0.8.5
>
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3023) NPE in describe_ring

Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085369#comment-13085369 ] 

Brandon Williams commented on CASSANDRA-3023:
---------------------------------------------

This will happen in a mixed cluster because nodes prior to CASSANDRA-1777 will not have advertised the information that describe_ring now relies on, causing the NPE when the nodes that do have 1777 try to access it.

There's not a lot we can do here.  We can basically return half-broken data for nodes that haven't advertised this info yet, but that puts us back to before CASSANDRA-1777, so it's not much of a win.  I can't think of a reason why clients would need to call describe_ring while there's an upgrade in process; topology isn't changing.

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3023) NPE in describe_ring

Posted by "Eric Falcao (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090472#comment-13090472 ] 

Eric Falcao commented on CASSANDRA-3023:
----------------------------------------

Just another quick note: by the comments here, I thought I'd be able to do a rolling restart to 0.8.4

By the time I got to my last node and stopped it, my clients were under the impression that the whole cluster was down (it wasn't) but checking cassandra logs I saw the stacktrace above spewing out 5-10 times per second. When the last node was started, the stacktraces were still flowing rapidly. I downgraded my entire cluster to 0.8.3 to stop the error and bring my site back online

Any insight?

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>             Fix For: 0.8.5
>
>         Attachments: 3023.txt
>
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-3023) NPE in describe_ring

Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brandon Williams updated CASSANDRA-3023:
----------------------------------------

    Attachment: 3023.txt

bq. I think "return half-broken data" is the right solution, since pre-1777 clients aren't going to be looking for the new data anyway.

You're right, but it sure bothers my OCD :)

Trivial patch attached.

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>             Fix For: 0.8.5
>
>         Attachments: 3023.txt
>
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-3023) NPE in describe_ring

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-3023:
--------------------------------------

    Fix Version/s: 0.8.5

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>             Fix For: 0.8.5
>
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3023) NPE in describe_ring

Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085353#comment-13085353 ] 

Brandon Williams commented on CASSANDRA-3023:
---------------------------------------------

Was the cluster still mixed between 0.8.3 and 0.8.4 at that point?  That would do it.  The easiest thing to do is not describe the ring until all machines are on the same version.

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3023) NPE in describe_ring

Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085309#comment-13085309 ] 

Brandon Williams commented on CASSANDRA-3023:
---------------------------------------------

This is going to happen anytime a) the tokens are known from being persisted in the system table and (so the coordinator knows about the nodes) but b) it has not actually talked to them, so it has no state information yet.  There's no good way to fix this, because either we leave out tokens that we know about but do not have any other information for, or we just leave the NPE.  On the bright side, the window to trigger this should be very short, probably just a handful of seconds at most.

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3023) NPE in describe_ring

Posted by "Eric Falcao (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085360#comment-13085360 ] 

Eric Falcao commented on CASSANDRA-3023:
----------------------------------------

i was upgrading one DC first and that's when i noticed it (and never upgraded the 2nd DC). Not sure if that matters. If so, I'll try upgrading both DC's and see if the error persists after fully upgraded.

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3023) NPE in describe_ring

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090321#comment-13090321 ] 

Jonathan Ellis commented on CASSANDRA-3023:
-------------------------------------------

+1

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>             Fix For: 0.8.5
>
>         Attachments: 3023.txt
>
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3023) NPE in describe_ring

Posted by "Wojciech Meler (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090140#comment-13090140 ] 

Wojciech Meler commented on CASSANDRA-3023:
-------------------------------------------

Describe_ring is needed even if topology isn't changing. Smart clients need to describe ring all the time - especially at startup to know which nodes they should contact. I have such implementation in C based client app.
Also Hadoop is describing ring prior to job execution in org.apache.cassandra.hadoop.ColumnFamilyInputFormat.

Will problem disappear after full upgrade to 0.8.4? Now I have work-around - call describe_ring only on my old 0.8.1 nodes, but after full upgrade if it won't work I will have a big problem...

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>             Fix For: 0.8.5
>
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-3023) NPE in describe_ring

Posted by "Eric Falcao (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085333#comment-13085333 ] 

Eric Falcao commented on CASSANDRA-3023:
----------------------------------------

FWIW, and I should have mentioned originally, the exception is thrown repeatedly every few seconds and doesn't seem to stop on the seed node

> NPE in describe_ring
> --------------------
>
>                 Key: CASSANDRA-3023
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3023
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.4
>            Reporter: Eric Falcao
>            Assignee: Brandon Williams
>
> Not sure how much of the following is relevant besides the stack trace, but here I go:
> I have a 2 DC, 2 node per DC cluster. DC1 had it's seed replaced but I hadn't restarted. I upgraded to 0.8.4 in the following fashion:
> -edited seeds
> -stopped both DC1 nodes
> -upgraded jars
> -started both nodes at the same time
> The non-seed node came up first and showed the following error. Then when the seed node came up, the error went away on the non-seed node but started occurring on the seed node:
> ERROR [pool-2-thread-15] 2011-08-12 22:32:27,438 Cassandra.java (line 3668) Internal error processing describe_ring
> java.lang.NullPointerException
> 	at org.apache.cassandra.service.StorageService.getRangeToRpcaddressMap(StorageService.java:623)
> 	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:731)
> 	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
> 	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
> 	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira