You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Aaron Morton (JIRA)" <ji...@apache.org> on 2012/05/07 01:10:48 UTC

[jira] [Created] (CASSANDRA-4223) Non Unique Streaming session ID's

Aaron Morton created CASSANDRA-4223:
---------------------------------------

             Summary: Non Unique Streaming session ID's
                 Key: CASSANDRA-4223
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 1.0.9
         Environment: Ubuntu 10.04.2 LTS

java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

"Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
The servers run on a custom hypervisor.
 
            Reporter: Aaron Morton
            Assignee: Aaron Morton


I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 

The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.

The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 

The errors were:

{code:java}
DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
        at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
        at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
{code}

and

{code:java}
DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
        at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
        at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
{code}


I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 

>From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 

{quote}
This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
{quote}

Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks

The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 

To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 

Hashing the file names and ranges is also a possibility, but more involved. 

(We may also want to drop latency times that are 0 nano seconds.)


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Morton updated CASSANDRA-4223:
------------------------------------

    Attachment: 4223_counter_session_id-V2.diff

4223_counter_session_id-V2.diff 

Uses stream source flag as discussed. Added the flags to StreamHeader so they were together. 

                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id-V2.diff, 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Morton updated CASSANDRA-4223:
------------------------------------

    Attachment: 4223_counter_session_id.diff

Use an AtomicLong in StreamInSession and one in StreamOutSession for the session id. 

Sessions are always accessed using <inet_address, session_id>, and in and out session are in their own collections. 
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274760#comment-13274760 ] 

Yuki Morishita commented on CASSANDRA-4223:
-------------------------------------------

I'm fine with Aaron's approach. Session Id counter will overflow if > Integer.MAX_VALUE but I think thats enough for streaming. Regarding the patch, doing 0 << 32 for StreamOutSession feels redundant to me. I think it's better to just increment the counter inside StreamOutSession constructor.
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id-V2.diff, 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-4223:
--------------------------------------

             Reviewer: yukim
    Affects Version/s:     (was: 1.0.9)
        Fix Version/s: 1.1.1
                       1.0.11
    
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271618#comment-13271618 ] 

Jonathan Ellis commented on CASSANDRA-4223:
-------------------------------------------

Right, so the problem is when A creates a {{<B, 1>}} Session in StreamIn, while B simultaneously creates a {{<B, 1>>}} for an unrelated StreamOut.transferRanges (move or unbootstrap).  
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277889#comment-13277889 ] 

Yuki Morishita commented on CASSANDRA-4223:
-------------------------------------------

Aaron,

Go ahead. :)
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id-V2.diff, 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272704#comment-13272704 ] 

Aaron Morton commented on CASSANDRA-4223:
-----------------------------------------

There are two path ways for a StreamOutSession to be added to the SOS.streams map:

1) Push / transfer. When node A wants to push data to node B using StreamOut.transferRanges(). In this case the SOS context using the target ip address - <node_b_ip, node_a_sos_counter>

2) Pull / request. When node B wants node A to send it data, using StreamIn.requestRanges(). Node B creates a StreamInSession that uses the target ip address - <node_a_ip, node_b_sis_counter>. When node A gets the stream request from node b it creates a StreamOutSession with context <node_b_ip, node_b_sis_counter>

So we can end up with these two sos contexts on node A:
* <node_b_ip, node_a_sos_counter>
* <node_b_ip, node_b_sis_counter>

We need to avoid collisions between node_a_sos_counter and node_b_sis_counter. It's an unlikely event but we need to be safe.

Adding the source_flag means we end up with this on node A:

* <node_b_ip, 1 "push", node_a_sos_counter>
* <node_b_ip, 0 "pull", node_b_sis_counter>


bq. Isn't it enough to just increment AtomicLong counter until there is no collision when creating locally?

Are you saying "lock around creating the StreamOutSessions from both code paths and create session ID's that do not collide with known session ids ?"

                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270838#comment-13270838 ] 

Jonathan Ellis commented on CASSANDRA-4223:
-------------------------------------------

It doesn't need to be unique-per-cluster because StreamInSession tracks it as a Pair<InetAddress, Long>.
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271577#comment-13271577 ] 

Jonathan Ellis commented on CASSANDRA-4223:
-------------------------------------------

StreamIn.requestRanges will create a session ID using the *target* node's id generator (whatever it is) but in a Pair with the *source* node's IP.

Conversely, when a Stream is originated on the source node, it creates a session id using the *source* node's id generator, and sends that over in the stream header, where IncomingStreamReader picks it up and sticks it in the Session map.

So it looks to me like Yuki was right the first time ... ?
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270545#comment-13270545 ] 

Jonathan Ellis commented on CASSANDRA-4223:
-------------------------------------------

... actually just an AtomicLong counter should be fine.  If we have more than 2^63 streaming sessions I'll live w/ the overflow. :)
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13275901#comment-13275901 ] 

Yuki Morishita commented on CASSANDRA-4223:
-------------------------------------------

OK, +1.
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id-V2.diff, 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270843#comment-13270843 ] 

Yuki Morishita commented on CASSANDRA-4223:
-------------------------------------------

I see possible collision here, since StreamOutSession is identified by destination host + timestamp(for now) and StreamInSession in destination node uses it when received it from remote.
Adding one more key (source IP?) would make ID unique?
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Morton updated CASSANDRA-4223:
------------------------------------

    Attachment: NanoTest.java

Test for unique nanoTime() results.
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.9
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>         Attachments: NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270071#comment-13270071 ] 

Jonathan Ellis commented on CASSANDRA-4223:
-------------------------------------------

bq. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 

Hector has (had?) some code to make per-client unique timestamps by taking counter = max(currentTimeMillis, counter + 1).  I suppose you could do something similar here so you don't need to busywait.
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.9
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>         Attachments: NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270076#comment-13270076 ] 

Aaron Morton commented on CASSANDRA-4223:
-----------------------------------------

yeah was thinking an incrementing counter.
Any concern about how unique the id's have to be ? 

I was thinking of creating a long out the (int) gossip generation for the local machine and an int counter that wraps around.   
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270886#comment-13270886 ] 

Jonathan Ellis commented on CASSANDRA-4223:
-------------------------------------------

Ugh, so the problem is that sometimes session IDs are generated by the target, and sometimes by the source?  That's broken...

I think there's three possible solutions:
# always generate session IDs on the source, so our Pair really is unique [the way I thought it worked :)]
# try to make the 64bit session IDs "unique enough" across the cluster [aaron's timestamp + counter]
# guarantee the session IDs are unique-per-host, and make the session context (source, id-generated-by-ip, id) instead of just (source, id) [yuki's suggestion]
# just switch to a UUID

#4 is probably simplest, but only #2 and #3 will be backwards-compatible.  Of those two I feel more confident about #3...  The only unique-id-in-64-bit schemas I know of require some coordination up front among participating nodes (e.g., http://engineering.twitter.com/2010/06/announcing-snowflake.html)
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271010#comment-13271010 ] 

Aaron Morton commented on CASSANDRA-4223:
-----------------------------------------

I'll take another look at how unique the session id need to be.
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271883#comment-13271883 ] 

Aaron Morton commented on CASSANDRA-4223:
-----------------------------------------

Cassandra 4226

Agree. We could end up with the StreamOutSessions with the same <ip, id> context, one generated externally one internally. 

bq. 3. guarantee the session IDs are unique-per-host, and make the session context (source, id-generated-by-ip, id) instead of just (source, id) [yuki's suggestion]

how does "id-generated-by-ip" work ? We can only generate a streaming context from the StreamHeader or Stream Reply and sender ip address. Unfortunately these messages are not extensible. 

Another idea: make the session id [source_flag + local_int_counter]. Source flag is:

* 0 for a StreamInSession generated to request data.
*  1 for a StreamOutSession generated to transfer/push data.
* A StreamOutSession created in response to a stream request (still) uses the sessionID generated remotely.

One node may now have multiple stream out sessions to the same ip with the same session id:

IP | source_flag | session id
1.1.1.1 | 0 (started remotely) | 6
1.1.1.1 | 1 (started locally) | 6   
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279068#comment-13279068 ] 

Aaron Morton commented on CASSANDRA-4223:
-----------------------------------------

committed to cassandra-1.0, cassandra-1.1 and trunk.

thanks. 
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id-V2.diff, 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Issue Comment Edited] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270886#comment-13270886 ] 

Jonathan Ellis edited comment on CASSANDRA-4223 at 5/8/12 10:12 PM:
--------------------------------------------------------------------

Ugh, so the problem is that sometimes session IDs are generated by the target, and sometimes by the source?  That's broken...

I think there's several possible solutions:
# always generate session IDs on the source, so our Pair really is unique [the way I thought it worked :)]
# try to make the 64bit session IDs "unique enough" across the cluster [aaron's timestamp + counter]
# guarantee the session IDs are unique-per-host, and make the session context (source, id-generated-by-ip, id) instead of just (source, id) [yuki's suggestion]
# just switch to a UUID

#4 is probably simplest, but only #2 and #3 will be backwards-compatible.  Of those two I feel more confident about #3...  The only unique-id-in-64-bit schemas I know of require some coordination up front among participating nodes (e.g., http://engineering.twitter.com/2010/06/announcing-snowflake.html)
                
      was (Author: jbellis):
    Ugh, so the problem is that sometimes session IDs are generated by the target, and sometimes by the source?  That's broken...

I think there's three possible solutions:
# always generate session IDs on the source, so our Pair really is unique [the way I thought it worked :)]
# try to make the 64bit session IDs "unique enough" across the cluster [aaron's timestamp + counter]
# guarantee the session IDs are unique-per-host, and make the session context (source, id-generated-by-ip, id) instead of just (source, id) [yuki's suggestion]
# just switch to a UUID

#4 is probably simplest, but only #2 and #3 will be backwards-compatible.  Of those two I feel more confident about #3...  The only unique-id-in-64-bit schemas I know of require some coordination up front among participating nodes (e.g., http://engineering.twitter.com/2010/06/announcing-snowflake.html)
                  
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270037#comment-13270037 ] 

Aaron Morton commented on CASSANDRA-4223:
-----------------------------------------

Did some more testing. On the stormondemand.com machines it looks like nanoTime() is updated every 10ms. On EC2 nodes nanoTime() was always unique. The test script is attached, examples are below...

{code}
root@db6:~/aaron# java -classpath ./ NanoTest 100
nanoTime 460519702418991 occurred 5176 times
nanoTime 460519712419053 occurred 12090 times
nanoTime 460519722419115 occurred 22602 times
nanoTime 460519732419177 occurred 36154 times
nanoTime 460519742419239 occurred 36089 times
nanoTime 460519752419301 occurred 36866 times
nanoTime 460519762419363 occurred 36997 times
nanoTime 460519772419425 occurred 36763 times
nanoTime 460519782419487 occurred 36910 times
nanoTime 460519792419549 occurred 35481 times
Ran for 100 milliseconds, got 295128 duplicates and 11 uniques.
{code}

If it takes 10ms to get a unique value calling multiple times is out of the question. Will put thinking hat back on.
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.9
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>         Attachments: fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271563#comment-13271563 ] 

Yuki Morishita commented on CASSANDRA-4223:
-------------------------------------------

After looking closer to the code, I can say that AtomicLong is enough here, as Aaron's patch suggests.
There is no need to generate cluster wide unique id, it just need to be unique between two nodes(source, dest).
I thought a pair of host and long value is shared among nodes but that's not true. My apologies.

So +1 to the patch attached.
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277722#comment-13277722 ] 

Aaron Morton commented on CASSANDRA-4223:
-----------------------------------------

Yuki were you going to commit this or do you want me to?
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id-V2.diff, 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271602#comment-13271602 ] 

Yuki Morishita commented on CASSANDRA-4223:
-------------------------------------------

Name *source* is probably misleading here. I found that it's actually *source* of data you're requesting.
When node A initiates streaming to node B with StreamIn.requestRanges, it creates StreamInSession with pair of id, say, <B, 1> and sends request with session id 1.
Node B receives request and creates StreamOutSession from sender's ip and received session id 1, ends up having StreamOutSession of id <A, 1>.

||A                    ||                              ||B                    ||
|StreamInSession<B, 1> | \-\-(session ID of 1 from A)\-\-> |StreamOutSession<A, 1> |
|                      | <\-(session ID of 1 from B)\-\-- |                       |
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270085#comment-13270085 ] 

Jonathan Ellis commented on CASSANDRA-4223:
-------------------------------------------

That sounds reasonable.  They just need to be unique-per-host, not per-cluster.
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270824#comment-13270824 ] 

Aaron Morton commented on CASSANDRA-4223:
-----------------------------------------

What about if a node received streaming requests from two other nodes? There would be an (small) chance of the nodes generating the same session ID.

Adding the gossip generation adds a little entropy to the id's. Happy to go with the simpler counter idea if you think this is a non-problem. 
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Morton updated CASSANDRA-4223:
------------------------------------

    Attachment: fmm streaming bug.txt
    
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.9
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>         Attachments: fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4223) Non Unique Streaming session ID's

Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272484#comment-13272484 ] 

Yuki Morishita commented on CASSANDRA-4223:
-------------------------------------------

Isn't it enough to just increment AtomicLong counter until there is no collision when creating locally?
                
> Non Unique Streaming session ID's
> ---------------------------------
>
>                 Key: CASSANDRA-4223
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4223
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 10.04.2 LTS
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> "Bare metal" servers from https://www.stormondemand.com/servers/baremetal.html 
> The servers run on a custom hypervisor.
>  
>            Reporter: Aaron Morton
>            Assignee: Aaron Morton
>              Labels: datastax_qa
>             Fix For: 1.0.11, 1.1.1
>
>         Attachments: 4223_counter_session_id.diff, NanoTest.java, fmm streaming bug.txt
>
>
> I have observed repair processes failing due to duplicate Streaming session ID's. In this installation it is preventing rebalance from completing. I believe it has also prevented repair from completing in the past. 
> The attached streaming-logs.txt file contains log messages and an explanation of what was happening during a repair operation. it has the evidence for duplicate session ID's.
> The duplicate session id's were generated on the repairing node and sent to the streaming node. The streaming source replaced the first session with the second which resulted in both sessions failing when the first FILE_COMPLETE message was received. 
> The errors were:
> {code:java}
> DEBUG [MiscStage:1] 2012-05-03 21:40:33,997 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:1] 2012-05-03 21:40:34,027 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:1,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/FMM_Studio/PartsData-hc-1-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> and
> {code:java}
> DEBUG [MiscStage:2] 2012-05-03 21:40:36,497 StreamReplyVerbHandler.java (line 47) Received StreamReply StreamReply(sessionId=26132848816442266, file='/var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db', action=FILE_FINISHED)
> ERROR [MiscStage:2] 2012-05-03 21:40:36,497 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MiscStage:2,5,main]
> java.lang.IllegalStateException: target reports current file is /var/lib/cassandra/data/OpsCenter/rollups7200-hc-3-Data.db but is null
>         at org.apache.cassandra.streaming.StreamOutSession.validateCurrentFile(StreamOutSession.java:195)
>         at org.apache.cassandra.streaming.StreamReplyVerbHandler.doVerb(StreamReplyVerbHandler.java:58)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {code}
> I think this is because System.nanoTime() is used for the session ID when creating the StreamInSession objects (driven from StorageService.requestRanges()) . 
> From the documentation (http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#nanoTime()) 
> {quote}
> This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. 
> {quote}
> Also some info here on clocks and timers https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> The hypervisor may be at fault here. But it seems like we cannot rely on successive calls to nanoTime() to return different values. 
> To avoid message/interface changes on the StreamHeader it would be good to keep the session ID a long. The simplest approach may be to make successive calls to nanoTime until the result changes. We could fail if a certain number of milliseconds have passed. 
> Hashing the file names and ranges is also a possibility, but more involved. 
> (We may also want to drop latency times that are 0 nano seconds.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira