You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (JIRA)" <ji...@apache.org> on 2012/06/27 21:01:44 UTC
[jira] [Created] (CASSANDRA-4384) HintedHandoff can begin before SS
knows the hostID
Brandon Williams created CASSANDRA-4384:
-------------------------------------------
Summary: HintedHandoff can begin before SS knows the hostID
Key: CASSANDRA-4384
URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2
Reporter: Brandon Williams
Fix For: 1.2
Since HH fires from the FD, SS won't quite have the hostId yet:
{noformat}
INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
java.lang.NullPointerException
at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4384) HintedHandoff can begin before SS
knows the hostID
Posted by "Eric Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Evans updated CASSANDRA-4384:
----------------------------------
Reviewer: brandon.williams
Assignee: Eric Evans (was: Brandon Williams)
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Eric Evans
> Fix For: 1.2
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Eric Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403994#comment-13403994 ]
Eric Evans commented on CASSANDRA-4384:
---------------------------------------
bq. ...so I would prefer this be in LocationInfo with the tokens...
Tokens are now in the {{peers}} columnfamily, keyed by the token with one column for the peer (inet address). With vnodes, that will become many-to-one (the endpoints will not be unique). Host IDs are one-to-one. The most straightforward approach seemed to be to create another columnfamily (peer_ids?), similar to {{peers}}, but keyed by host ID.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Eric Evans
> Fix For: 1.2
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476952#comment-13476952 ]
Sylvain Lebresne commented on CASSANDRA-4384:
---------------------------------------------
We do persist the ring_id for all nodes now, so maybe we should use that.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.0 beta 1
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.2.0
>
> Attachments: 4384.txt
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411655#comment-13411655 ]
Brandon Williams commented on CASSANDRA-4384:
---------------------------------------------
I like making HOST_ID a full app state, that seems much cleaner. Unsure on the ip vs uuid change at the moment.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Eric Evans
> Fix For: 1.2
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404003#comment-13404003 ]
Jonathan Ellis commented on CASSANDRA-4384:
-------------------------------------------
or we could replace the existing {{peers}} entirely (I'd prefer to retain the table name) and add an index on the token column for that lookup.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Eric Evans
> Fix For: 1.2
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447930#comment-13447930 ]
Brandon Williams commented on CASSANDRA-4384:
---------------------------------------------
HOST_ID is a full app state after CASSANDRA-4383 now so we can revisit this.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.0 beta 1
> Reporter: Brandon Williams
> Assignee: Eric Evans
> Fix For: 1.2.0
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404006#comment-13404006 ]
Sylvain Lebresne commented on CASSANDRA-4384:
---------------------------------------------
bq. or we could replace the existing peers entirely
I would agree with that, but going a bit further I would imagine that a natural schema for that peers table could be something like:
{noformat}
CREATE TABLE peers (
host_id UUID PRIMARY KEY,
peer inet,
tokens set<blob>,
)
{noformat}
with maybe even more info for each host but that's for CASSANDRA-4351. Of course that specific schema means waiting for CASSANDRA-3647, but I don't see that as a big problem. This wouldn't let use query by token but I don't think that really matter because we'll read the whole table content at startup anyway.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Eric Evans
> Fix For: 1.2
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498168#comment-13498168 ]
Jonathan Ellis commented on CASSANDRA-4384:
-------------------------------------------
+1 on Brandon's fix for 1.2.0.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.0 beta 1
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.2.0 rc1
>
> Attachments: 4384.txt
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403315#comment-13403315 ]
Sylvain Lebresne commented on CASSANDRA-4384:
---------------------------------------------
CASSANDRA-4351 is relevant (or in other words, I agree too since we wanted to persist them anyway for that issue).
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.2
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Sam Overton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411623#comment-13411623 ]
Sam Overton commented on CASSANDRA-4384:
----------------------------------------
I have a branch at https://github.com/acunu/cassandra/tree/CASSANDRA-4384 which
* gossips host ID on startup
* persists host IDs to the peers CF and loads them on startup
I ended up changing the peers schema to be keyed off IP address instead of UUID:
{noformat}
CREATE TABLE peers (
peer inet PRIMARY KEY,
token_bytes blob,
ring_id uuid,
)
{noformat}
I agree with Sylvain that it seems more natural to use UUID as the key rather than by IP, but that didn't seem as straightforward to update when tokens or hostIds change.
There is some overlap with CASSANDRA-4122 so it would be best to wait for that to go in before changing the SystemTable again for this.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Eric Evans
> Fix For: 1.2
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4384) HintedHandoff can begin before SS
knows the hostID
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-4384:
----------------------------------------
Attachment: 4384.txt
Not sure where we stand on the schema changes, but we can trivially prevent this exception by checking the gossiper instead of SS now.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.0 beta 1
> Reporter: Brandon Williams
> Assignee: Eric Evans
> Fix For: 1.2.0
>
> Attachments: 4384.txt
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reassigned CASSANDRA-4384:
-------------------------------------------
Assignee: Brandon Williams
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.2
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403966#comment-13403966 ]
Brandon Williams commented on CASSANDRA-4384:
---------------------------------------------
I'll note that while CASSANDRA-4351 is similar in spirit, that is information we do NOT want to use, so I would prefer this be in LocationInfo with the tokens, and have everything else in a different CF, so the old method of blowing away LI when something goes bad is still valid.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Eric Evans
> Fix For: 1.2
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Sam Overton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403132#comment-13403132 ]
Sam Overton commented on CASSANDRA-4384:
----------------------------------------
When a node first starts up without a hostId map, it may be asked to store a hint for a node which is down and it won't know the hostId of that node since it never saw it alive.
We should persist the hostIds to prevent both of these cases where we are missing required information when we first start up.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.2
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403309#comment-13403309 ]
Brandon Williams commented on CASSANDRA-4384:
---------------------------------------------
I'm inclined to agree with Sam, since this is how the problem is solved with tokens.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.2
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4384) HintedHandoff can begin before
SS knows the hostID
Posted by "Eric Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402628#comment-13402628 ]
Eric Evans commented on CASSANDRA-4384:
---------------------------------------
This is a corner-case that only happens to a node that is restarted (and lost its hostId map as a result). It is the result of a hint delivery being triggered (by the _reception_ of a gossip message) before the hostId could be processed from the message.
I see two ways to fix:
# Skip hint delivery when we don't (yet) have a hostId.
# Persist hostIds the way we do tokens.
(1) has the benefit of being a one-liner. Presumably this code exists to schedule hint delivery for a remote host that was dead and is now alive. Since in this case it is _us_ that was down, I don't think skipping would be Evil.
(2) adds complexity, but guards against any future cases of an {{IEndpointStateChangeSubscriber.onAlive()}} relying on {{TokenMetadata.isMember()}} before looking up a hostId.
> HintedHandoff can begin before SS knows the hostID
> --------------------------------------------------
>
> Key: CASSANDRA-4384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4384
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.2
>
>
> Since HH fires from the FD, SS won't quite have the hostId yet:
> {noformat}
> INFO 18:58:04,196 Started hinted handoff for host: null with IP: /10.179.65.102
> INFO 18:58:04,197 Node /10.179.65.102 state jump to normal
> ERROR 18:58:04,197 Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:120)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:87)
> at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:433)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Simple solution seems to be getting the hostId from gossip instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira