You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by "Dr. Martin Grabmüller" <Ma...@eleven.de> on 2010/05/03 09:30:55 UTC

Strange streaming behaviour

Hello list,

I encountered a problem with streaming in my test cluster but
found nothing like this in JIRA or on the list.

I'm running a test cluster of three nodes, RF=3, Cassandra 0.6.1.
I started the first node and inserted some data, then bootstrapped
the other machines one after the other.  No problems here.

Then, I inserted data over the weekend and checked back this morning,
finding a lot of -tmp-files in the data directory of the two bootstrapepd
machines which seem to be leftovers from streaming.

Interesting thing is, they are mostly all about the same size on each node. Additionally,
this size is the same as the size of some file to be streamed (see the output of
nodetool streams at the end of this post):

[Node 2:]

-rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 07:39 AccountList-tmp-7-Data.db
-rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 00:39 AddressList-tmp-7-Data.db
-rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 07:53 IndexQueue-tmp-7-Data.db
-rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 05:19 MailMetadata-tmp-7-Data.db
-rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 00:53 Statistics-tmp-7-Data.db

[Node 3:]

-rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 07:39 AccountList-tmp-6-Data.db
-rw-r--r-- 1 cassandra cassandra       325 2010-05-03 00:39 AddressList-tmp-6-Data.db
-rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 08:02 CustomerList-tmp-6-Data.db
-rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 07:53 IndexQueue-tmp-7-Data.db
-rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 01:02 MailMetadata-tmp-7-Data.db
-rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 00:53 Statistics-tmp-7-Data.db

Checking the logs, I find a few streaming-related error messages:

[Node 1 (yes, the node without leftover files):]

ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-02 14:39:10,675 DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
java.lang.RuntimeException: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-Data.db (No such file or directory)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-Data.db (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
        at org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84)
        at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:63)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more
ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-02 14:39:10,675 CassandraDaemon.java (line 78) Fatal exception in thread Thread[MESSAGE-STREAMING-POOL:1,5,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-Data.db (No such file or directory)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-Data.db (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
        at org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84)
        at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:63)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more

[Node 3:]

ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-03 00:39:46,656 DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
java.lang.RuntimeException: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db (No such file or directory)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
        at org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84)
        at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:63)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more
ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-03 00:39:46,656 CassandraDaemon.java (line 78) Fatal exception in thread Thread[MESSAGE-STREAMING-POOL:1,5,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db (No such file or directory)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
        at org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84)
        at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:63)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more

On Node 2, there are no errors.

I have not been able to reproduce this error with the default configuration.

Has anybody an idea how to debug this problem?  Is this known?  Should I create
a ticket?

Thanks in advance!

Martin

P.S. Here is the output of nodetool streams for the three nodes:

[Node 1:] 192.168.211.44

Mode: Normal
 Nothing streaming to /192.168.211.46
 Nothing streaming to /192.168.211.45
Not receiving any streams.

[Node 2:] 192.168.211.45

Mode: Normal
Streaming to: /192.168.211.46
   /mnt/data000/cassandra/data/Archive/stream/AddressList-6-Data.db 10994/10994
   /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Index.db 0/3142
   /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Data.db 0/15974
   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Index.db 0/758284
   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Filter.db 0/16525
   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Data.db 0/5581529
   /mnt/data000/cassandra/data/Archive/stream/AccountList-6-Index.db 0/592
   /mnt/data000/cassandra/data/Archive/stream/AccountList-6-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/AccountList-6-Data.db 0/2866
   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Index.db 0/648
   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Data.db 0/8801484
   /mnt/data000/cassandra/data/Archive/stream/CustomerList-6-Index.db 0/61
   /mnt/data000/cassandra/data/Archive/stream/CustomerList-6-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/CustomerList-6-Data.db 0/229
   /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Index.db 0/12171
   /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Filter.db 0/445
   /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Data.db 0/9148876
   /mnt/data000/cassandra/data/Archive/stream/MailList-88-Index.db 0/59273
   /mnt/data000/cassandra/data/Archive/stream/MailList-88-Filter.db 0/36685
   /mnt/data000/cassandra/data/Archive/stream/MailList-88-Data.db 0/109059024
   /mnt/data000/cassandra/data/Archive/stream/Headers-11-Index.db 0/632998
   /mnt/data000/cassandra/data/Archive/stream/Headers-11-Filter.db 0/24685
   /mnt/data000/cassandra/data/Archive/stream/Headers-11-Data.db 0/5148176
   /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Index.db 0/2360
   /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Data.db 0/10994
   /mnt/data000/cassandra/data/Archive/stream/Statistics-12-Index.db 0/3142
   /mnt/data000/cassandra/data/Archive/stream/Statistics-12-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/Statistics-12-Data.db 0/15996
   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Index.db 0/789717
   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Filter.db 0/29005
   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Data.db 0/5791834
   /mnt/data000/cassandra/data/Archive/stream/AccountList-12-Index.db 0/592
   /mnt/data000/cassandra/data/Archive/stream/AccountList-12-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/AccountList-12-Data.db 0/2866
   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Index.db 0/648
   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Data.db 0/5280403
   /mnt/data000/cassandra/data/Archive/stream/CustomerList-12-Index.db 0/61
   /mnt/data000/cassandra/data/Archive/stream/CustomerList-12-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/CustomerList-12-Data.db 0/229
   /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Index.db 0/12171
   /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Filter.db 0/445
   /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Data.db 0/14228834
 Nothing streaming to /192.168.211.44
Streaming from: /192.168.211.46
   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-7-Filter.db 0/325
   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-7-Data.db 0/229
   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Index.db 0/12171
   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Filter.db 0/445
   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Data.db 0/9050963
   Archive: /mnt/data000/cassandra/data/Archive/MailList-tmp-87-Index.db 0/58383
   Archive: /mnt/data000/cassandra/data/Archive/MailList-tmp-87-Filter.db 0/36685
   Archive: /mnt/data000/cassandra/data/Archive/MailList-tmp-87-Data.db 0/107327613
   Archive: /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Index.db 0/646210
   Archive: /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Filter.db 0/24925
   Archive: /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Data.db 0/5254628
   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Index.db 0/2360
   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Filter.db 0/325
   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Data.db 0/10994
   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-13-Index.db 0/3142
   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-13-Filter.db 0/325
   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-13-Data.db 0/15995
   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Index.db 0/802581
   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Filter.db 0/29125
   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Data.db 0/5885615
   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-13-Index.db 0/592
   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-13-Filter.db 0/325
   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-13-Data.db 0/2866
   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Index.db 0/648
   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Filter.db 0/325
   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Data.db 0/5310585
   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-13-Index.db 0/61
   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-13-Filter.db 0/325
   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-13-Data.db 0/229
   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Index.db 0/12171
   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Filter.db 0/445
   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Data.db 0/14129376
   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-7-Index.db 2360/3142
   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Data.db 2360/5673882
   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-7-Data.db 2360/2866
   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Filter.db 2360/16645
   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-7-Data.db 2360/8830815
   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-7-Data.db 2360/10994
   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-7-Data.db 2360/15971
   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Index.db 2360/770984

[Node 3:] 192.168.211.46

Mode: Normal
 Nothing streaming to /192.168.211.44
Streaming to: /192.168.211.45
   /mnt/data000/cassandra/data/Archive/stream/AddressList-7-Index.db 2360/2360
   /mnt/data000/cassandra/data/Archive/stream/AddressList-7-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/AddressList-7-Data.db 0/10994
   /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Index.db 0/3142
   /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Data.db 0/15971
   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Index.db 0/770984
   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Filter.db 0/16645
   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Data.db 0/5673882
   /mnt/data000/cassandra/data/Archive/stream/AccountList-7-Index.db 0/592
   /mnt/data000/cassandra/data/Archive/stream/AccountList-7-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/AccountList-7-Data.db 0/2866
   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Index.db 0/648
   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Data.db 0/8830815
   /mnt/data000/cassandra/data/Archive/stream/CustomerList-7-Index.db 0/61
   /mnt/data000/cassandra/data/Archive/stream/CustomerList-7-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/CustomerList-7-Data.db 0/229
   /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Index.db 0/12171
   /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Filter.db 0/445
   /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Data.db 0/9050963
   /mnt/data000/cassandra/data/Archive/stream/MailList-86-Index.db 0/58383
   /mnt/data000/cassandra/data/Archive/stream/MailList-86-Filter.db 0/36685
   /mnt/data000/cassandra/data/Archive/stream/MailList-86-Data.db 0/107327613
   /mnt/data000/cassandra/data/Archive/stream/Headers-11-Index.db 0/646210
   /mnt/data000/cassandra/data/Archive/stream/Headers-11-Filter.db 0/24925
   /mnt/data000/cassandra/data/Archive/stream/Headers-11-Data.db 0/5254628
   /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Index.db 0/2360
   /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Data.db 0/10994
   /mnt/data000/cassandra/data/Archive/stream/Statistics-13-Index.db 0/3142
   /mnt/data000/cassandra/data/Archive/stream/Statistics-13-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/Statistics-13-Data.db 0/15995
   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Index.db 0/802581
   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Filter.db 0/29125
   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Data.db 0/5885615
   /mnt/data000/cassandra/data/Archive/stream/AccountList-13-Index.db 0/592
   /mnt/data000/cassandra/data/Archive/stream/AccountList-13-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/AccountList-13-Data.db 0/2866
   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Index.db 0/648
   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Data.db 0/5310585
   /mnt/data000/cassandra/data/Archive/stream/CustomerList-13-Index.db 0/61
   /mnt/data000/cassandra/data/Archive/stream/CustomerList-13-Filter.db 0/325
   /mnt/data000/cassandra/data/Archive/stream/CustomerList-13-Data.db 0/229
   /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Index.db 0/12171
   /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Filter.db 0/445
   /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Data.db 0/14129376
Streaming from: /192.168.211.45
   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Index.db 0/12171
   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Filter.db 0/445
   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Data.db 0/9148876
   Archive: /mnt/data000/cassandra/data/Archive/MailList-tmp-88-Index.db 0/59273
   Archive: /mnt/data000/cassandra/data/Archive/MailList-tmp-88-Filter.db 0/36685
   Archive: /mnt/data000/cassandra/data/Archive/MailList-tmp-88-Data.db 0/109059024
   Archive: /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Index.db 0/632998
   Archive: /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Filter.db 0/24685
   Archive: /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Data.db 0/5148176
   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Index.db 0/2360
   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Filter.db 0/325
   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Data.db 0/10994
   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-12-Index.db 0/3142
   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-12-Filter.db 0/325
   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-12-Data.db 0/15996
   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Index.db 0/789717
   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Filter.db 0/29005
   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Data.db 0/5791834
   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-12-Index.db 0/592
   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-12-Filter.db 0/325
   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-12-Data.db 0/2866
   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Index.db 0/648
   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Filter.db 0/325
   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Data.db 0/5280403
   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-12-Index.db 0/61
   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-12-Filter.db 0/325
   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-12-Data.db 0/229
   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Index.db 0/12171
   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Filter.db 0/445
   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Data.db 0/14228834
   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Index.db 10994/758284
   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-6-Data.db 325/10994
   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-7-Data.db 10994/8801484
   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Data.db 10994/5581529
   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Filter.db 10994/16525
   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-7-Data.db 10994/15974

RE: Strange streaming behaviour

Posted by "Dr. Martin Grabmüller" <Ma...@eleven.de>.

Gary,

thanks for the feedback.

Today, I tried to reproduce, but have not been able to.  I successfully
bootstrapped two new nodes into my cluster while the cluster was
under load, but streaming did not get stuck this time.

Maybe somehow my local setup is responsible or my original diagnostics
was not correct.

I will create a ticket when I have been able to reproduce.

Unfortunately, running with debug logging enabled is too slow to
load gigabytes of data into a cluster in a sensible time frame.

Martin 

> -----Original Message-----
> From: Gary Dusbabek [mailto:gdusbabek@gmail.com] 
> Sent: Monday, May 03, 2010 5:57 PM
> To: user@cassandra.apache.org
> Subject: Re: Strange streaming behaviour
> 
> Martin,
> 
> Please create a ticket and include the relevant parts of your 
> storage-conf.
> 
> To summarize, the output gives you the impression that bootstrap has
> completed normally, but when you check, it appears to be hung on the
> receiving nodes?
> 
> Do you mind turning debug on and and seeing if you can reproduce?  The
> strange part is that the source node doesn't think it's streaming at
> all.
> 
> Gary.
> 
> 
> On Mon, May 3, 2010 at 02:30, Dr. Martin Grabmüller
> <Ma...@eleven.de> wrote:
> > Hello list,
> >
> > I encountered a problem with streaming in my test cluster but
> > found nothing like this in JIRA or on the list.
> >
> > I'm running a test cluster of three nodes, RF=3, Cassandra 0.6.1.
> > I started the first node and inserted some data, then bootstrapped
> > the other machines one after the other.  No problems here.
> >
> > Then, I inserted data over the weekend and checked back 
> this morning,
> > finding a lot of -tmp-files in the data directory of the 
> two bootstrapepd
> > machines which seem to be leftovers from streaming.
> >
> > Interesting thing is, they are mostly all about the same 
> size on each node. Additionally,
> > this size is the same as the size of some file to be 
> streamed (see the output of
> > nodetool streams at the end of this post):
> >
> > [Node 2:]
> >
> > -rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 07:39 
> AccountList-tmp-7-Data.db
> > -rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 00:39 
> AddressList-tmp-7-Data.db
> > -rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 07:53 
> IndexQueue-tmp-7-Data.db
> > -rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 05:19 
> MailMetadata-tmp-7-Data.db
> > -rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 00:53 
> Statistics-tmp-7-Data.db
> >
> > [Node 3:]
> >
> > -rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 07:39 
> AccountList-tmp-6-Data.db
> > -rw-r--r-- 1 cassandra cassandra       325 2010-05-03 00:39 
> AddressList-tmp-6-Data.db
> > -rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 08:02 
> CustomerList-tmp-6-Data.db
> > -rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 07:53 
> IndexQueue-tmp-7-Data.db
> > -rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 01:02 
> MailMetadata-tmp-7-Data.db
> > -rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 00:53 
> Statistics-tmp-7-Data.db
> >
> > Checking the logs, I find a few streaming-related error messages:
> >
> > [Node 1 (yes, the node without leftover files):]
> >
> > ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-02 14:39:10,675 
> DebuggableThreadPoolExecutor.java (line 101) Error in 
> ThreadPoolExecutor
> > java.lang.RuntimeException: java.io.FileNotFoundException: 
> /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-
> Data.db (No such file or directory)
> >        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable
> .java:34)
> >        at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx
> ecutor.java:1110)
> >        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE
> xecutor.java:603)
> >        at java.lang.Thread.run(Thread.java:636)
> > Caused by: java.io.FileNotFoundException: 
> /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-
> Data.db (No such file or directory)
> >        at java.io.RandomAccessFile.open(Native Method)
> >        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> >        at 
> org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84)
> >        at 
> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStream
> Task.java:63)
> >        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable
> .java:30)
> >        ... 3 more
> > ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-02 14:39:10,675 
> CassandraDaemon.java (line 78) Fatal exception in thread 
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> > java.lang.RuntimeException: java.io.FileNotFoundException: 
> /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-
> Data.db (No such file or directory)
> >        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable
> .java:34)
> >        at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx
> ecutor.java:1110)
> >        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE
> xecutor.java:603)
> >        at java.lang.Thread.run(Thread.java:636)
> > Caused by: java.io.FileNotFoundException: 
> /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-
> Data.db (No such file or directory)
> >        at java.io.RandomAccessFile.open(Native Method)
> >        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> >        at 
> org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84)
> >        at 
> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStream
> Task.java:63)
> >        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable
> .java:30)
> >        ... 3 more
> >
> > [Node 3:]
> >
> > ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-03 00:39:46,656 
> DebuggableThreadPoolExecutor.java (line 101) Error in 
> ThreadPoolExecutor
> > java.lang.RuntimeException: java.io.FileNotFoundException: 
> /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db 
> (No such file or directory)
> >        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable
> .java:34)
> >        at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx
> ecutor.java:1110)
> >        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE
> xecutor.java:603)
> >        at java.lang.Thread.run(Thread.java:636)
> > Caused by: java.io.FileNotFoundException: 
> /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db 
> (No such file or directory)
> >        at java.io.RandomAccessFile.open(Native Method)
> >        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> >        at 
> org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84)
> >        at 
> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStream
> Task.java:63)
> >        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable
> .java:30)
> >        ... 3 more
> > ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-03 00:39:46,656 
> CassandraDaemon.java (line 78) Fatal exception in thread 
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> > java.lang.RuntimeException: java.io.FileNotFoundException: 
> /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db 
> (No such file or directory)
> >        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable
> .java:34)
> >        at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx
> ecutor.java:1110)
> >        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE
> xecutor.java:603)
> >        at java.lang.Thread.run(Thread.java:636)
> > Caused by: java.io.FileNotFoundException: 
> /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db 
> (No such file or directory)
> >        at java.io.RandomAccessFile.open(Native Method)
> >        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> >        at 
> org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84)
> >        at 
> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStream
> Task.java:63)
> >        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable
> .java:30)
> >        ... 3 more
> >
> > On Node 2, there are no errors.
> >
> > I have not been able to reproduce this error with the 
> default configuration.
> >
> > Has anybody an idea how to debug this problem?  Is this 
> known?  Should I create
> > a ticket?
> >
> > Thanks in advance!
> >
> > Martin
> >
> > P.S. Here is the output of nodetool streams for the three nodes:
> >
> > [Node 1:] 192.168.211.44
> >
> > Mode: Normal
> >  Nothing streaming to /192.168.211.46
> >  Nothing streaming to /192.168.211.45
> > Not receiving any streams.
> >
> > [Node 2:] 192.168.211.45
> >
> > Mode: Normal
> > Streaming to: /192.168.211.46
> >   
> /mnt/data000/cassandra/data/Archive/stream/AddressList-6-Data.
> db 10994/10994
> >   
> /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Index.
> db 0/3142
> >   
> /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Filter
> .db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Data.d
> b 0/15974
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Inde
> x.db 0/758284
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Filt
> er.db 0/16525
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Data
> .db 0/5581529
> >   
> /mnt/data000/cassandra/data/Archive/stream/AccountList-6-Index
> .db 0/592
> >   
> /mnt/data000/cassandra/data/Archive/stream/AccountList-6-Filte
> r.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/AccountList-6-Data.
> db 0/2866
> >   
> /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Index.db 0/648
> >   
> /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Filter
> .db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Data.d
> b 0/8801484
> >   
> /mnt/data000/cassandra/data/Archive/stream/CustomerList-6-Inde
> x.db 0/61
> >   
> /mnt/data000/cassandra/data/Archive/stream/CustomerList-6-Filt
> er.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/CustomerList-6-Data
> .db 0/229
> >   
> /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Index.db 0/12171
> >   
> /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Filter.db 0/445
> >   
> /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Data.db 
> 0/9148876
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailList-88-Index.d
> b 0/59273
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailList-88-Filter.
> db 0/36685
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailList-88-Data.db
>  0/109059024
> >   
> /mnt/data000/cassandra/data/Archive/stream/Headers-11-Index.db
>  0/632998
> >   
> /mnt/data000/cassandra/data/Archive/stream/Headers-11-Filter.d
> b 0/24685
> >   
> /mnt/data000/cassandra/data/Archive/stream/Headers-11-Data.db 
> 0/5148176
> >   
> /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Inde
> x.db 0/2360
> >   
> /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Filt
> er.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Data
> .db 0/10994
> >   
> /mnt/data000/cassandra/data/Archive/stream/Statistics-12-Index
> .db 0/3142
> >   
> /mnt/data000/cassandra/data/Archive/stream/Statistics-12-Filte
> r.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/Statistics-12-Data.
> db 0/15996
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Ind
> ex.db 0/789717
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Fil
> ter.db 0/29005
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Dat
> a.db 0/5791834
> >   
> /mnt/data000/cassandra/data/Archive/stream/AccountList-12-Inde
> x.db 0/592
> >   
> /mnt/data000/cassandra/data/Archive/stream/AccountList-12-Filt
> er.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/AccountList-12-Data
> .db 0/2866
> >   
> /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Index
> .db 0/648
> >   
> /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Filte
> r.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Data.
> db 0/5280403
> >   
> /mnt/data000/cassandra/data/Archive/stream/CustomerList-12-Ind
> ex.db 0/61
> >   
> /mnt/data000/cassandra/data/Archive/stream/CustomerList-12-Fil
> ter.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/CustomerList-12-Dat
> a.db 0/229
> >   
> /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Index.d
> b 0/12171
> >   
> /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Filter.db 0/445
> >   
> /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Data.db
>  0/14228834
> >  Nothing streaming to /192.168.211.44
> > Streaming from: /192.168.211.46
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/CustomerList-tmp-7-Filter.db 0/325
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/CustomerList-tmp-7-Data.db 0/229
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Index.db 0/12171
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Filter.db 0/445
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Data.db 0/9050963
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailList-tmp-87-Index.db 0/58383
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailList-tmp-87-Filter.db 0/36685
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailList-tmp-87-Data.db 
> 0/107327613
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Index.db 0/646210
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Filter.db 0/24925
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Data.db 0/5254628
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Index.db 0/2360
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Filter.db 0/325
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Data.db 0/10994
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Statistics-tmp-13-Index.db 0/3142
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Statistics-tmp-13-Filter.db 0/325
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Statistics-tmp-13-Data.db 0/15995
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Index.
> db 0/802581
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Filter
> .db 0/29125
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Data.d
> b 0/5885615
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AccountList-tmp-13-Index.db 0/592
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AccountList-tmp-13-Filter.db 0/325
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AccountList-tmp-13-Data.db 0/2866
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Index.db 0/648
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Filter.db 0/325
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Data.db 
> 0/5310585
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/CustomerList-tmp-13-Index.db 0/61
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/CustomerList-tmp-13-Filter
> .db 0/325
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/CustomerList-tmp-13-Data.db 0/229
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Index.db 0/12171
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Filter.db 0/445
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Data.db 0/14129376
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Statistics-tmp-7-Index.db 
> 2360/3142
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Data.db
>  2360/5673882
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AccountList-tmp-7-Data.db 
> 2360/2866
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Filter.
> db 2360/16645
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-7-Data.db 
> 2360/8830815
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AddressList-tmp-7-Data.db 
> 2360/10994
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Statistics-tmp-7-Data.db 
> 2360/15971
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Index.d
> b 2360/770984
> >
> > [Node 3:] 192.168.211.46
> >
> > Mode: Normal
> >  Nothing streaming to /192.168.211.44
> > Streaming to: /192.168.211.45
> >   
> /mnt/data000/cassandra/data/Archive/stream/AddressList-7-Index
> .db 2360/2360
> >   
> /mnt/data000/cassandra/data/Archive/stream/AddressList-7-Filte
> r.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/AddressList-7-Data.
> db 0/10994
> >   
> /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Index.
> db 0/3142
> >   
> /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Filter
> .db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Data.d
> b 0/15971
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Inde
> x.db 0/770984
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Filt
> er.db 0/16645
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Data
> .db 0/5673882
> >   
> /mnt/data000/cassandra/data/Archive/stream/AccountList-7-Index
> .db 0/592
> >   
> /mnt/data000/cassandra/data/Archive/stream/AccountList-7-Filte
> r.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/AccountList-7-Data.
> db 0/2866
> >   
> /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Index.db 0/648
> >   
> /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Filter
> .db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Data.d
> b 0/8830815
> >   
> /mnt/data000/cassandra/data/Archive/stream/CustomerList-7-Inde
> x.db 0/61
> >   
> /mnt/data000/cassandra/data/Archive/stream/CustomerList-7-Filt
> er.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/CustomerList-7-Data
> .db 0/229
> >   
> /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Index.db 0/12171
> >   
> /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Filter.db 0/445
> >   
> /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Data.db 
> 0/9050963
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailList-86-Index.d
> b 0/58383
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailList-86-Filter.
> db 0/36685
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailList-86-Data.db
>  0/107327613
> >   
> /mnt/data000/cassandra/data/Archive/stream/Headers-11-Index.db
>  0/646210
> >   
> /mnt/data000/cassandra/data/Archive/stream/Headers-11-Filter.d
> b 0/24925
> >   
> /mnt/data000/cassandra/data/Archive/stream/Headers-11-Data.db 
> 0/5254628
> >   
> /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Inde
> x.db 0/2360
> >   
> /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Filt
> er.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Data
> .db 0/10994
> >   
> /mnt/data000/cassandra/data/Archive/stream/Statistics-13-Index
> .db 0/3142
> >   
> /mnt/data000/cassandra/data/Archive/stream/Statistics-13-Filte
> r.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/Statistics-13-Data.
> db 0/15995
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Ind
> ex.db 0/802581
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Fil
> ter.db 0/29125
> >   
> /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Dat
> a.db 0/5885615
> >   
> /mnt/data000/cassandra/data/Archive/stream/AccountList-13-Inde
> x.db 0/592
> >   
> /mnt/data000/cassandra/data/Archive/stream/AccountList-13-Filt
> er.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/AccountList-13-Data
> .db 0/2866
> >   
> /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Index
> .db 0/648
> >   
> /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Filte
> r.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Data.
> db 0/5310585
> >   
> /mnt/data000/cassandra/data/Archive/stream/CustomerList-13-Ind
> ex.db 0/61
> >   
> /mnt/data000/cassandra/data/Archive/stream/CustomerList-13-Fil
> ter.db 0/325
> >   
> /mnt/data000/cassandra/data/Archive/stream/CustomerList-13-Dat
> a.db 0/229
> >   
> /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Index.d
> b 0/12171
> >   
> /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Filter.db 0/445
> >   
> /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Data.db
>  0/14129376
> > Streaming from: /192.168.211.45
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Index.db 0/12171
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Filter.db 0/445
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Data.db 0/9148876
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailList-tmp-88-Index.db 0/59273
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailList-tmp-88-Filter.db 0/36685
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailList-tmp-88-Data.db 
> 0/109059024
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Index.db 0/632998
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Filter.db 0/24685
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Data.db 0/5148176
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Index.db 0/2360
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Filter.db 0/325
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Data.db 0/10994
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Statistics-tmp-12-Index.db 0/3142
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Statistics-tmp-12-Filter.db 0/325
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Statistics-tmp-12-Data.db 0/15996
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Index.
> db 0/789717
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Filter
> .db 0/29005
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Data.d
> b 0/5791834
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AccountList-tmp-12-Index.db 0/592
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AccountList-tmp-12-Filter.db 0/325
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AccountList-tmp-12-Data.db 0/2866
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Index.db 0/648
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Filter.db 0/325
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Data.db 
> 0/5280403
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/CustomerList-tmp-12-Index.db 0/61
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/CustomerList-tmp-12-Filter
> .db 0/325
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/CustomerList-tmp-12-Data.db 0/229
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Index.db 0/12171
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Filter.db 0/445
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Data.db 0/14228834
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Index.d
> b 10994/758284
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/AddressList-tmp-6-Data.db 
> 325/10994
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-7-Data.db 
> 10994/8801484
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Data.db
>  10994/5581529
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Filter.
> db 10994/16525
> >   Archive: 
> /mnt/data000/cassandra/data/Archive/Statistics-tmp-7-Data.db 
> 10994/15974
> >
>

Re: Strange streaming behaviour

Posted by Gary Dusbabek <gd...@gmail.com>.

Martin,

Please create a ticket and include the relevant parts of your storage-conf.

To summarize, the output gives you the impression that bootstrap has
completed normally, but when you check, it appears to be hung on the
receiving nodes?

Do you mind turning debug on and and seeing if you can reproduce?  The
strange part is that the source node doesn't think it's streaming at
all.

Gary.


On Mon, May 3, 2010 at 02:30, Dr. Martin Grabmüller
<Ma...@eleven.de> wrote:
> Hello list,
>
> I encountered a problem with streaming in my test cluster but
> found nothing like this in JIRA or on the list.
>
> I'm running a test cluster of three nodes, RF=3, Cassandra 0.6.1.
> I started the first node and inserted some data, then bootstrapped
> the other machines one after the other.  No problems here.
>
> Then, I inserted data over the weekend and checked back this morning,
> finding a lot of -tmp-files in the data directory of the two bootstrapepd
> machines which seem to be leftovers from streaming.
>
> Interesting thing is, they are mostly all about the same size on each node. Additionally,
> this size is the same as the size of some file to be streamed (see the output of
> nodetool streams at the end of this post):
>
> [Node 2:]
>
> -rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 07:39 AccountList-tmp-7-Data.db
> -rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 00:39 AddressList-tmp-7-Data.db
> -rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 07:53 IndexQueue-tmp-7-Data.db
> -rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 05:19 MailMetadata-tmp-7-Data.db
> -rw-r--r-- 1 cassandra cassandra      2360 2010-05-03 00:53 Statistics-tmp-7-Data.db
>
> [Node 3:]
>
> -rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 07:39 AccountList-tmp-6-Data.db
> -rw-r--r-- 1 cassandra cassandra       325 2010-05-03 00:39 AddressList-tmp-6-Data.db
> -rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 08:02 CustomerList-tmp-6-Data.db
> -rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 07:53 IndexQueue-tmp-7-Data.db
> -rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 01:02 MailMetadata-tmp-7-Data.db
> -rw-r--r-- 1 cassandra cassandra     10994 2010-05-03 00:53 Statistics-tmp-7-Data.db
>
> Checking the logs, I find a few streaming-related error messages:
>
> [Node 1 (yes, the node without leftover files):]
>
> ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-02 14:39:10,675 DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-Data.db (No such file or directory)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-Data.db (No such file or directory)
>        at java.io.RandomAccessFile.open(Native Method)
>        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>        at org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84)
>        at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:63)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 3 more
> ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-02 14:39:10,675 CassandraDaemon.java (line 78) Fatal exception in thread Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-Data.db (No such file or directory)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-Data.db (No such file or directory)
>        at java.io.RandomAccessFile.open(Native Method)
>        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>        at org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84)
>        at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:63)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 3 more
>
> [Node 3:]
>
> ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-03 00:39:46,656 DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db (No such file or directory)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db (No such file or directory)
>        at java.io.RandomAccessFile.open(Native Method)
>        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>        at org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84)
>        at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:63)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 3 more
> ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-03 00:39:46,656 CassandraDaemon.java (line 78) Fatal exception in thread Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db (No such file or directory)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db (No such file or directory)
>        at java.io.RandomAccessFile.open(Native Method)
>        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>        at org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84)
>        at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:63)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 3 more
>
> On Node 2, there are no errors.
>
> I have not been able to reproduce this error with the default configuration.
>
> Has anybody an idea how to debug this problem?  Is this known?  Should I create
> a ticket?
>
> Thanks in advance!
>
> Martin
>
> P.S. Here is the output of nodetool streams for the three nodes:
>
> [Node 1:] 192.168.211.44
>
> Mode: Normal
>  Nothing streaming to /192.168.211.46
>  Nothing streaming to /192.168.211.45
> Not receiving any streams.
>
> [Node 2:] 192.168.211.45
>
> Mode: Normal
> Streaming to: /192.168.211.46
>   /mnt/data000/cassandra/data/Archive/stream/AddressList-6-Data.db 10994/10994
>   /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Index.db 0/3142
>   /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Data.db 0/15974
>   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Index.db 0/758284
>   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Filter.db 0/16525
>   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Data.db 0/5581529
>   /mnt/data000/cassandra/data/Archive/stream/AccountList-6-Index.db 0/592
>   /mnt/data000/cassandra/data/Archive/stream/AccountList-6-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/AccountList-6-Data.db 0/2866
>   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Index.db 0/648
>   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Data.db 0/8801484
>   /mnt/data000/cassandra/data/Archive/stream/CustomerList-6-Index.db 0/61
>   /mnt/data000/cassandra/data/Archive/stream/CustomerList-6-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/CustomerList-6-Data.db 0/229
>   /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Index.db 0/12171
>   /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Filter.db 0/445
>   /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Data.db 0/9148876
>   /mnt/data000/cassandra/data/Archive/stream/MailList-88-Index.db 0/59273
>   /mnt/data000/cassandra/data/Archive/stream/MailList-88-Filter.db 0/36685
>   /mnt/data000/cassandra/data/Archive/stream/MailList-88-Data.db 0/109059024
>   /mnt/data000/cassandra/data/Archive/stream/Headers-11-Index.db 0/632998
>   /mnt/data000/cassandra/data/Archive/stream/Headers-11-Filter.db 0/24685
>   /mnt/data000/cassandra/data/Archive/stream/Headers-11-Data.db 0/5148176
>   /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Index.db 0/2360
>   /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Data.db 0/10994
>   /mnt/data000/cassandra/data/Archive/stream/Statistics-12-Index.db 0/3142
>   /mnt/data000/cassandra/data/Archive/stream/Statistics-12-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/Statistics-12-Data.db 0/15996
>   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Index.db 0/789717
>   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Filter.db 0/29005
>   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Data.db 0/5791834
>   /mnt/data000/cassandra/data/Archive/stream/AccountList-12-Index.db 0/592
>   /mnt/data000/cassandra/data/Archive/stream/AccountList-12-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/AccountList-12-Data.db 0/2866
>   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Index.db 0/648
>   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Data.db 0/5280403
>   /mnt/data000/cassandra/data/Archive/stream/CustomerList-12-Index.db 0/61
>   /mnt/data000/cassandra/data/Archive/stream/CustomerList-12-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/CustomerList-12-Data.db 0/229
>   /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Index.db 0/12171
>   /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Filter.db 0/445
>   /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Data.db 0/14228834
>  Nothing streaming to /192.168.211.44
> Streaming from: /192.168.211.46
>   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-7-Filter.db 0/325
>   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-7-Data.db 0/229
>   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Index.db 0/12171
>   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Filter.db 0/445
>   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Data.db 0/9050963
>   Archive: /mnt/data000/cassandra/data/Archive/MailList-tmp-87-Index.db 0/58383
>   Archive: /mnt/data000/cassandra/data/Archive/MailList-tmp-87-Filter.db 0/36685
>   Archive: /mnt/data000/cassandra/data/Archive/MailList-tmp-87-Data.db 0/107327613
>   Archive: /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Index.db 0/646210
>   Archive: /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Filter.db 0/24925
>   Archive: /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Data.db 0/5254628
>   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Index.db 0/2360
>   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Filter.db 0/325
>   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Data.db 0/10994
>   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-13-Index.db 0/3142
>   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-13-Filter.db 0/325
>   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-13-Data.db 0/15995
>   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Index.db 0/802581
>   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Filter.db 0/29125
>   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Data.db 0/5885615
>   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-13-Index.db 0/592
>   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-13-Filter.db 0/325
>   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-13-Data.db 0/2866
>   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Index.db 0/648
>   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Filter.db 0/325
>   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Data.db 0/5310585
>   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-13-Index.db 0/61
>   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-13-Filter.db 0/325
>   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-13-Data.db 0/229
>   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Index.db 0/12171
>   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Filter.db 0/445
>   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Data.db 0/14129376
>   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-7-Index.db 2360/3142
>   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Data.db 2360/5673882
>   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-7-Data.db 2360/2866
>   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Filter.db 2360/16645
>   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-7-Data.db 2360/8830815
>   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-7-Data.db 2360/10994
>   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-7-Data.db 2360/15971
>   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Index.db 2360/770984
>
> [Node 3:] 192.168.211.46
>
> Mode: Normal
>  Nothing streaming to /192.168.211.44
> Streaming to: /192.168.211.45
>   /mnt/data000/cassandra/data/Archive/stream/AddressList-7-Index.db 2360/2360
>   /mnt/data000/cassandra/data/Archive/stream/AddressList-7-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/AddressList-7-Data.db 0/10994
>   /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Index.db 0/3142
>   /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Data.db 0/15971
>   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Index.db 0/770984
>   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Filter.db 0/16645
>   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Data.db 0/5673882
>   /mnt/data000/cassandra/data/Archive/stream/AccountList-7-Index.db 0/592
>   /mnt/data000/cassandra/data/Archive/stream/AccountList-7-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/AccountList-7-Data.db 0/2866
>   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Index.db 0/648
>   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Data.db 0/8830815
>   /mnt/data000/cassandra/data/Archive/stream/CustomerList-7-Index.db 0/61
>   /mnt/data000/cassandra/data/Archive/stream/CustomerList-7-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/CustomerList-7-Data.db 0/229
>   /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Index.db 0/12171
>   /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Filter.db 0/445
>   /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Data.db 0/9050963
>   /mnt/data000/cassandra/data/Archive/stream/MailList-86-Index.db 0/58383
>   /mnt/data000/cassandra/data/Archive/stream/MailList-86-Filter.db 0/36685
>   /mnt/data000/cassandra/data/Archive/stream/MailList-86-Data.db 0/107327613
>   /mnt/data000/cassandra/data/Archive/stream/Headers-11-Index.db 0/646210
>   /mnt/data000/cassandra/data/Archive/stream/Headers-11-Filter.db 0/24925
>   /mnt/data000/cassandra/data/Archive/stream/Headers-11-Data.db 0/5254628
>   /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Index.db 0/2360
>   /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Data.db 0/10994
>   /mnt/data000/cassandra/data/Archive/stream/Statistics-13-Index.db 0/3142
>   /mnt/data000/cassandra/data/Archive/stream/Statistics-13-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/Statistics-13-Data.db 0/15995
>   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Index.db 0/802581
>   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Filter.db 0/29125
>   /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Data.db 0/5885615
>   /mnt/data000/cassandra/data/Archive/stream/AccountList-13-Index.db 0/592
>   /mnt/data000/cassandra/data/Archive/stream/AccountList-13-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/AccountList-13-Data.db 0/2866
>   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Index.db 0/648
>   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Data.db 0/5310585
>   /mnt/data000/cassandra/data/Archive/stream/CustomerList-13-Index.db 0/61
>   /mnt/data000/cassandra/data/Archive/stream/CustomerList-13-Filter.db 0/325
>   /mnt/data000/cassandra/data/Archive/stream/CustomerList-13-Data.db 0/229
>   /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Index.db 0/12171
>   /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Filter.db 0/445
>   /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Data.db 0/14129376
> Streaming from: /192.168.211.45
>   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Index.db 0/12171
>   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Filter.db 0/445
>   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Data.db 0/9148876
>   Archive: /mnt/data000/cassandra/data/Archive/MailList-tmp-88-Index.db 0/59273
>   Archive: /mnt/data000/cassandra/data/Archive/MailList-tmp-88-Filter.db 0/36685
>   Archive: /mnt/data000/cassandra/data/Archive/MailList-tmp-88-Data.db 0/109059024
>   Archive: /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Index.db 0/632998
>   Archive: /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Filter.db 0/24685
>   Archive: /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Data.db 0/5148176
>   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Index.db 0/2360
>   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Filter.db 0/325
>   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Data.db 0/10994
>   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-12-Index.db 0/3142
>   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-12-Filter.db 0/325
>   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-12-Data.db 0/15996
>   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Index.db 0/789717
>   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Filter.db 0/29005
>   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Data.db 0/5791834
>   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-12-Index.db 0/592
>   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-12-Filter.db 0/325
>   Archive: /mnt/data000/cassandra/data/Archive/AccountList-tmp-12-Data.db 0/2866
>   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Index.db 0/648
>   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Filter.db 0/325
>   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Data.db 0/5280403
>   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-12-Index.db 0/61
>   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-12-Filter.db 0/325
>   Archive: /mnt/data000/cassandra/data/Archive/CustomerList-tmp-12-Data.db 0/229
>   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Index.db 0/12171
>   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Filter.db 0/445
>   Archive: /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Data.db 0/14228834
>   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Index.db 10994/758284
>   Archive: /mnt/data000/cassandra/data/Archive/AddressList-tmp-6-Data.db 325/10994
>   Archive: /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-7-Data.db 10994/8801484
>   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Data.db 10994/5581529
>   Archive: /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Filter.db 10994/16525
>   Archive: /mnt/data000/cassandra/data/Archive/Statistics-tmp-7-Data.db 10994/15974
>