You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Yatong Zhang <bl...@gmail.com> on 2014/04/29 11:09:40 UTC
bugs report for 2.0.7
Hi there,
Sorry if this is not the right place to report bugs. I am using 2.0.7 and I
have a 10 boxes clusters with about 200TB capacity. I just found I had 3
boxes with error exceptions. With datastax opscenter I can see these three
nodes lost connections (no reponse), but after I sshed to these server,
cassandara were still running, and the 'system.log' still had logs.
I think this might be a bug so any one would kindly help to investigate
into it? Thanks~
ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249 CassandraDaemon.java
> (line 198) Exception in thread Thread[CompactionExecutor:1,1,main]
> FSWriteError in /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db
> at
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475)
> at
> org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.write(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:295)
> at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
> at
> org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34)
> at
> org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)
> at
> org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41)
> at
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:468)
> ... 13 more
> ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406 StorageService.java
> (line 367) Stopping gossiper
> WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406 StorageService.java
> (line 281) Stopping gossip by operator request
> INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java (line
> 1271) Announcing shutdown
> ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406 StorageService.java
> (line 372) Stopping RPC server
> INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406 ThriftServer.java
> (line 141) Stop listening to thrift clients
> ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417 StorageService.java
> (line 377) Stopping native transport
> INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java (line
> 181) Stop listening for CQL clients
>
Re: bugs report for 2.0.7
Posted by Yatong Zhang <bl...@gmail.com>.
It looks like Cassandra will calculate free space by summing all the disks,
and this will result in a big compaction file that can not fit in one disk.
So any one could clarify this please?
On Wed, Apr 30, 2014 at 2:09 PM, Yatong Zhang <bl...@gmail.com> wrote:
> I am using CQL 3 to create a table to store images and very image was
> about 200K ~ 500K. I have 6 harddisks per node and cassandra was configured
> with 6 data directories:
>
> data_file_directories:
>> - /data1/cass
>> - /data2/cass
>> - /data3/cass
>> - /data4/cass
>> - /data5/cass
>> - /data6/cass
>>
>
> And every directory is on a standalone disk. But I just found when the
> error occurred:
>
> [root@node5 images]# ll -hl
>> total 3.6T
>> drwxr-xr-x 4 root root 4.0K Jan 20 09:44 snapshots
>> -rw-r--r-- 1 root root 456M Apr 30 13:42
>> mydb-images-tmp-jb-91068-CompressionInfo.db
>> -rw-r--r-- 1 root root 3.5T Apr 30 13:42 mydb-images-tmp-jb-91068-Data.db
>> -rw-r--r-- 1 root root 0 Apr 30 13:42
>> mydb-images-tmp-jb-91068-Filter.db
>> -rw-r--r-- 1 root root 2.0G Apr 30 13:42 mydb-images-tmp-jb-91068-Index.db
>>
>
> [root@node5 images]# df -hl
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 49G 7.5G 39G 17% /
> tmpfs 7.8G 0 7.8G 0% /dev/shm
> /dev/sda3 3.6T 1.3T 2.1T 38% /data1
> /dev/sdb1 3.6T 1.4T 2.1T 39% /data2
> /dev/sdc1 3.6T 466G 3.0T 14% /data3
> /dev/sdd1 3.6T 1.3T 2.2T 38% /data4
> /dev/sde1 3.6T 1.3T 2.2T 38% /data5
> /dev/sdf1 3.6T 3.6T 0 100% /data6
>
> *mydb-images-tmp-jb-91068-Data.db *almost occupied all the disk space (4T
> harddisk with 3.6T actual usable size)
>
> after I restated cassandra, very thing seems to be fine:
>
> -rw-r--r-- 1 root root 19K Apr 30 13:58
>> mydb_oe-images-tmp-jb-96242-CompressionInfo.db
>> -rw-r--r-- 1 root root 145M Apr 30 13:58
>> mydb_oe-images-tmp-jb-96242-Data.db
>> -rw-r--r-- 1 root root 64K Apr 30 13:58
>> mydb_oe-images-tmp-jb-96242-Index.db
>>
>
> [root@node5 images]# df -hl
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 49G 7.5G 39G 17% /
> tmpfs 7.8G 0 7.8G 0% /dev/shm
> /dev/sda3 3.6T 1.3T 2.1T 38% /data1
> /dev/sdb1 3.6T 1.4T 2.1T 39% /data2
> /dev/sdc1 3.6T 466G 3.0T 14% /data3
> /dev/sdd1 3.6T 1.3T 2.2T 38% /data4
> /dev/sde1 3.6T 1.3T 2.2T 38% /data5
> /dev/sdf1 3.6T 662M 3.4T 1% /data6
>
> So my questions are:
>
> 1. I am using CQL3 and is there a limit for 'tables' created by CQL3?
> 2. I specified 6 data directories with each on a stand alone disk, is it
> OK?
> 3. Why the tmp db file is so large? Is it normal or a bug?
>
>
> So could any one please help to solve this issue? Any help is of great
> appreciation and thanks a lot!
>
>
> On Wed, Apr 30, 2014 at 12:04 PM, Yatong Zhang <bl...@gmail.com>wrote:
>
>> Thanks for the response. I've checked the system logs and harddisk smartd
>> info, and no errors found. Any hints to locate the problem?
>>
>>
>> On Wed, Apr 30, 2014 at 9:26 AM, Michael Shuler <mi...@pbandjelly.org>wrote:
>>
>>> Then you likely need to fix your I/O problem. The most recent error you
>>> posted is an EOFException - the file being read ended unexpectedly.
>>> Probably when you ran out of disk space.
>>>
>>> --
>>> Michael
>>>
>>>
>>> On 04/29/2014 07:48 PM, Yatong Zhang wrote:
>>>
>>>> Here is another type of exception, seems all are I/O related:
>>>>
>>>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,548 SSTableReader.java
>>>> (line
>>>>
>>>>> 223) Opening
>>>>> /data2/cass/system/compaction_history/system-compaction_
>>>>> history-jb-6956
>>>>> (447252 bytes)
>>>>> INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,553 SSTableReader.java
>>>>> (line 223) Opening
>>>>> /data2/cass/system/compaction_history/system-compaction_
>>>>> history-jb-6958
>>>>> (257 bytes)
>>>>> INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,554 SSTableReader.java
>>>>> (line 223) Opening
>>>>> /data2/cass/system/compaction_history/system-compaction_
>>>>> history-jb-6957
>>>>> (257 bytes)
>>>>> INFO [main] 2014-04-29 14:44:35,592 ColumnFamilyStore.java (line 248)
>>>>> Initializing system.batchlog
>>>>> INFO [main] 2014-04-29 14:44:35,596 ColumnFamilyStore.java (line 248)
>>>>> Initializing system.sstable_activity
>>>>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,601 SSTableReader.java
>>>>> (line 223) Opening
>>>>> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8084
>>>>> (1562
>>>>> bytes)
>>>>> INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,604 SSTableReader.java
>>>>> (line 223) Opening
>>>>> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8083
>>>>> (2075
>>>>> bytes)
>>>>> INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,605 SSTableReader.java
>>>>> (line 223) Opening
>>>>> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8085
>>>>> (1555
>>>>> bytes)
>>>>> INFO [main] 2014-04-29 14:44:35,687 AutoSavingCache.java (line 114)
>>>>> reading saved cache
>>>>> /data1/saved_caches/system-sstable_activity-KeyCache-b.db
>>>>> INFO [main] 2014-04-29 14:44:35,696 ColumnFamilyStore.java (line 248)
>>>>> Initializing system.peer_events
>>>>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,697 SSTableReader.java
>>>>> (line 223) Opening /data4/cass/system/peer_
>>>>> events/system-peer_events-jb-181
>>>>> (12342 bytes)
>>>>> INFO [main] 2014-04-29 14:44:35,717 ColumnFamilyStore.java (line 248)
>>>>> Initializing system.compactions_in_progress
>>>>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,718 SSTableReader.java
>>>>> (line 223) Opening
>>>>> /data5/cass/system/compactions_in_progress/system-compactions_in_
>>>>> progress-jb-36448
>>>>> (167 bytes)
>>>>> ERROR [SSTableBatchOpen:1] 2014-04-29 14:44:35,730 CassandraDaemon.java
>>>>> (line 198) Exception in thread Thread[SSTableBatchOpen:1,5,main]
>>>>> org.apache.cassandra.io.sstable.CorruptSSTableException:
>>>>> java.io.EOFException
>>>>> at
>>>>> org.apache.cassandra.io.compress.CompressionMetadata.<
>>>>> init>(CompressionMetadata.java:110)
>>>>> at
>>>>> org.apache.cassandra.io.compress.CompressionMetadata.
>>>>> create(CompressionMetadata.java:64)
>>>>> at
>>>>> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile
>>>>> $Builder.complete(CompressedPoolingSegmentedFile.java:42)
>>>>> at
>>>>> org.apache.cassandra.io.sstable.SSTableReader.load(
>>>>> SSTableReader.java:458)
>>>>> at
>>>>> org.apache.cassandra.io.sstable.SSTableReader.load(
>>>>> SSTableReader.java:422)
>>>>> at
>>>>> org.apache.cassandra.io.sstable.SSTableReader.open(
>>>>> SSTableReader.java:203)
>>>>> at
>>>>> org.apache.cassandra.io.sstable.SSTableReader.open(
>>>>> SSTableReader.java:184)
>>>>> at
>>>>> org.apache.cassandra.io.sstable.SSTableReader$1.run(
>>>>> SSTableReader.java:264)
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>> call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>> ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>> ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> Caused by: java.io.EOFException
>>>>> at
>>>>> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
>>>>> at java.io.DataInputStream.readUTF(DataInputStream.java:589)
>>>>> at java.io.DataInputStream.readUTF(DataInputStream.java:564)
>>>>> at
>>>>> org.apache.cassandra.io.compress.CompressionMetadata.<
>>>>> init>(CompressionMetadata.java:85)
>>>>> ... 12 more
>>>>> INFO [main] 2014-04-29 14:44:35,733 ColumnFamilyStore.java (line 248)
>>>>> Initializing system.hints
>>>>> INFO [main] 2014-04-29 14:44:35,734 AutoSavingCache.java (line 114)
>>>>> reading saved cache /data1/saved_caches/system-hints-KeyCache-b.db
>>>>> INFO [main] 2014-04-29 14:44:35,737 ColumnFamilyStore.java (line 248)
>>>>> Initializing system.schema_keyspaces
>>>>>
>>>>>
>>>>
>>>>
>>>> On Tue, Apr 29, 2014 at 6:07 PM, Yatong Zhang <bl...@gmail.com>
>>>> wrote:
>>>>
>>>> I am pretty sure the disk has plenty of space, I am sure of that. I
>>>>> restarted cassandra and everything went fine again.
>>>>>
>>>>> It's really wired
>>>>>
>>>>>
>>>>> On Tue, Apr 29, 2014 at 5:58 PM, Sylvain Lebresne <
>>>>> sylvain@datastax.com>wrote:
>>>>>
>>>>> The important part of that stack trace is "java.io.IOException: No
>>>>>> space
>>>>>> left on device", your disks are full (and it's not really a bug that
>>>>>> Cassandra error out in that case).
>>>>>>
>>>>>> --
>>>>>> Sylvain
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 29, 2014 at 11:09 AM, Yatong Zhang <bl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi there,
>>>>>>>
>>>>>>> Sorry if this is not the right place to report bugs. I am using 2.0.7
>>>>>>>
>>>>>> and I
>>>>>>
>>>>>>> have a 10 boxes clusters with about 200TB capacity. I just found I
>>>>>>> had 3
>>>>>>> boxes with error exceptions. With datastax opscenter I can see these
>>>>>>>
>>>>>> three
>>>>>>
>>>>>>> nodes lost connections (no reponse), but after I sshed to these
>>>>>>> server,
>>>>>>> cassandara were still running, and the 'system.log' still had logs.
>>>>>>>
>>>>>>> I think this might be a bug so any one would kindly help to
>>>>>>> investigate
>>>>>>> into it? Thanks~
>>>>>>>
>>>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249
>>>>>>>
>>>>>> CassandraDaemon.java
>>>>>>
>>>>>>> (line 198) Exception in thread Thread[CompactionExecutor:1,1,main]
>>>>>>>> FSWriteError in
>>>>>>>>
>>>>>>> /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db
>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(
>>>>>> SSTableWriter.java:475)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cassandra.io.util.FileUtils.closeQuietly(
>>>>>> FileUtils.java:212)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>> org.apache.cassandra.io.sstable.SSTableWriter.abort(
>>>>>> SSTableWriter.java:301)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>> org.apache.cassandra.db.compaction.CompactionTask.
>>>>>> runWith(CompactionTask.java:209)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(
>>>>>> DiskAwareRunnable.java:48)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cassandra.utils.WrappedRunnable.run(
>>>>>> WrappedRunnable.java:28)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(
>>>>>> CompactionTask.java:60)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(
>>>>>> AbstractCompactionTask.java:59)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>> org.apache.cassandra.db.compaction.CompactionManager$
>>>>>> BackgroundCompactionTask.run(CompactionManager.java:197)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>> call(Executors.java:471)
>>>>>>
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>> ThreadPoolExecutor.java:615)
>>>>>>
>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Caused by: java.io.IOException: No space left on device
>>>>>>>> at java.io.FileOutputStream.write(Native Method)
>>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:
>>>>>>>> 295)
>>>>>>>> at
>>>>>>>>
>>>>>>> java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>> org.apache.cassandra.utils.BloomFilterSerializer.serialize(
>>>>>> BloomFilterSerializer.java:34)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>> org.apache.cassandra.utils.Murmur3BloomFilter$
>>>>>> Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cassandra.utils.FilterFactory.serialize(
>>>>>> FilterFactory.java:41)
>>>>>>
>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(
>>>>>> SSTableWriter.java:468)
>>>>>>
>>>>>>> ... 13 more
>>>>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406
>>>>>>>>
>>>>>>> StorageService.java
>>>>>>
>>>>>>> (line 367) Stopping gossiper
>>>>>>>> WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406
>>>>>>>>
>>>>>>> StorageService.java
>>>>>>
>>>>>>> (line 281) Stopping gossip by operator request
>>>>>>>> INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java
>>>>>>>>
>>>>>>> (line
>>>>>>
>>>>>>> 1271) Announcing shutdown
>>>>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406
>>>>>>>>
>>>>>>> StorageService.java
>>>>>>
>>>>>>> (line 372) Stopping RPC server
>>>>>>>> INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406
>>>>>>>> ThriftServer.java
>>>>>>>> (line 141) Stop listening to thrift clients
>>>>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417
>>>>>>>>
>>>>>>> StorageService.java
>>>>>>
>>>>>>> (line 377) Stopping native transport
>>>>>>>> INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java
>>>>>>>> (line
>>>>>>>> 181) Stop listening for CQL clients
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: bugs report for 2.0.7
Posted by Yatong Zhang <bl...@gmail.com>.
I am using CQL 3 to create a table to store images and very image was about
200K ~ 500K. I have 6 harddisks per node and cassandra was configured with
6 data directories:
data_file_directories:
> - /data1/cass
> - /data2/cass
> - /data3/cass
> - /data4/cass
> - /data5/cass
> - /data6/cass
>
And every directory is on a standalone disk. But I just found when the
error occurred:
[root@node5 images]# ll -hl
> total 3.6T
> drwxr-xr-x 4 root root 4.0K Jan 20 09:44 snapshots
> -rw-r--r-- 1 root root 456M Apr 30 13:42
> mydb-images-tmp-jb-91068-CompressionInfo.db
> -rw-r--r-- 1 root root 3.5T Apr 30 13:42 mydb-images-tmp-jb-91068-Data.db
> -rw-r--r-- 1 root root 0 Apr 30 13:42 mydb-images-tmp-jb-91068-Filter.db
> -rw-r--r-- 1 root root 2.0G Apr 30 13:42 mydb-images-tmp-jb-91068-Index.db
>
[root@node5 images]# df -hl
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 49G 7.5G 39G 17% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda3 3.6T 1.3T 2.1T 38% /data1
/dev/sdb1 3.6T 1.4T 2.1T 39% /data2
/dev/sdc1 3.6T 466G 3.0T 14% /data3
/dev/sdd1 3.6T 1.3T 2.2T 38% /data4
/dev/sde1 3.6T 1.3T 2.2T 38% /data5
/dev/sdf1 3.6T 3.6T 0 100% /data6
*mydb-images-tmp-jb-91068-Data.db *almost occupied all the disk space (4T
harddisk with 3.6T actual usable size)
after I restated cassandra, very thing seems to be fine:
-rw-r--r-- 1 root root 19K Apr 30 13:58
> mydb_oe-images-tmp-jb-96242-CompressionInfo.db
> -rw-r--r-- 1 root root 145M Apr 30 13:58
> mydb_oe-images-tmp-jb-96242-Data.db
> -rw-r--r-- 1 root root 64K Apr 30 13:58
> mydb_oe-images-tmp-jb-96242-Index.db
>
[root@node5 images]# df -hl
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 49G 7.5G 39G 17% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda3 3.6T 1.3T 2.1T 38% /data1
/dev/sdb1 3.6T 1.4T 2.1T 39% /data2
/dev/sdc1 3.6T 466G 3.0T 14% /data3
/dev/sdd1 3.6T 1.3T 2.2T 38% /data4
/dev/sde1 3.6T 1.3T 2.2T 38% /data5
/dev/sdf1 3.6T 662M 3.4T 1% /data6
So my questions are:
1. I am using CQL3 and is there a limit for 'tables' created by CQL3?
2. I specified 6 data directories with each on a stand alone disk, is it OK?
3. Why the tmp db file is so large? Is it normal or a bug?
So could any one please help to solve this issue? Any help is of great
appreciation and thanks a lot!
On Wed, Apr 30, 2014 at 12:04 PM, Yatong Zhang <bl...@gmail.com> wrote:
> Thanks for the response. I've checked the system logs and harddisk smartd
> info, and no errors found. Any hints to locate the problem?
>
>
> On Wed, Apr 30, 2014 at 9:26 AM, Michael Shuler <mi...@pbandjelly.org>wrote:
>
>> Then you likely need to fix your I/O problem. The most recent error you
>> posted is an EOFException - the file being read ended unexpectedly.
>> Probably when you ran out of disk space.
>>
>> --
>> Michael
>>
>>
>> On 04/29/2014 07:48 PM, Yatong Zhang wrote:
>>
>>> Here is another type of exception, seems all are I/O related:
>>>
>>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,548 SSTableReader.java
>>> (line
>>>
>>>> 223) Opening
>>>> /data2/cass/system/compaction_history/system-compaction_history-jb-6956
>>>> (447252 bytes)
>>>> INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,553 SSTableReader.java
>>>> (line 223) Opening
>>>> /data2/cass/system/compaction_history/system-compaction_history-jb-6958
>>>> (257 bytes)
>>>> INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,554 SSTableReader.java
>>>> (line 223) Opening
>>>> /data2/cass/system/compaction_history/system-compaction_history-jb-6957
>>>> (257 bytes)
>>>> INFO [main] 2014-04-29 14:44:35,592 ColumnFamilyStore.java (line 248)
>>>> Initializing system.batchlog
>>>> INFO [main] 2014-04-29 14:44:35,596 ColumnFamilyStore.java (line 248)
>>>> Initializing system.sstable_activity
>>>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,601 SSTableReader.java
>>>> (line 223) Opening
>>>> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8084
>>>> (1562
>>>> bytes)
>>>> INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,604 SSTableReader.java
>>>> (line 223) Opening
>>>> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8083
>>>> (2075
>>>> bytes)
>>>> INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,605 SSTableReader.java
>>>> (line 223) Opening
>>>> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8085
>>>> (1555
>>>> bytes)
>>>> INFO [main] 2014-04-29 14:44:35,687 AutoSavingCache.java (line 114)
>>>> reading saved cache
>>>> /data1/saved_caches/system-sstable_activity-KeyCache-b.db
>>>> INFO [main] 2014-04-29 14:44:35,696 ColumnFamilyStore.java (line 248)
>>>> Initializing system.peer_events
>>>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,697 SSTableReader.java
>>>> (line 223) Opening /data4/cass/system/peer_
>>>> events/system-peer_events-jb-181
>>>> (12342 bytes)
>>>> INFO [main] 2014-04-29 14:44:35,717 ColumnFamilyStore.java (line 248)
>>>> Initializing system.compactions_in_progress
>>>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,718 SSTableReader.java
>>>> (line 223) Opening
>>>> /data5/cass/system/compactions_in_progress/system-compactions_in_
>>>> progress-jb-36448
>>>> (167 bytes)
>>>> ERROR [SSTableBatchOpen:1] 2014-04-29 14:44:35,730 CassandraDaemon.java
>>>> (line 198) Exception in thread Thread[SSTableBatchOpen:1,5,main]
>>>> org.apache.cassandra.io.sstable.CorruptSSTableException:
>>>> java.io.EOFException
>>>> at
>>>> org.apache.cassandra.io.compress.CompressionMetadata.<
>>>> init>(CompressionMetadata.java:110)
>>>> at
>>>> org.apache.cassandra.io.compress.CompressionMetadata.
>>>> create(CompressionMetadata.java:64)
>>>> at
>>>> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile
>>>> $Builder.complete(CompressedPoolingSegmentedFile.java:42)
>>>> at
>>>> org.apache.cassandra.io.sstable.SSTableReader.load(
>>>> SSTableReader.java:458)
>>>> at
>>>> org.apache.cassandra.io.sstable.SSTableReader.load(
>>>> SSTableReader.java:422)
>>>> at
>>>> org.apache.cassandra.io.sstable.SSTableReader.open(
>>>> SSTableReader.java:203)
>>>> at
>>>> org.apache.cassandra.io.sstable.SSTableReader.open(
>>>> SSTableReader.java:184)
>>>> at
>>>> org.apache.cassandra.io.sstable.SSTableReader$1.run(
>>>> SSTableReader.java:264)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>> ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>> ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> Caused by: java.io.EOFException
>>>> at
>>>> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
>>>> at java.io.DataInputStream.readUTF(DataInputStream.java:589)
>>>> at java.io.DataInputStream.readUTF(DataInputStream.java:564)
>>>> at
>>>> org.apache.cassandra.io.compress.CompressionMetadata.<
>>>> init>(CompressionMetadata.java:85)
>>>> ... 12 more
>>>> INFO [main] 2014-04-29 14:44:35,733 ColumnFamilyStore.java (line 248)
>>>> Initializing system.hints
>>>> INFO [main] 2014-04-29 14:44:35,734 AutoSavingCache.java (line 114)
>>>> reading saved cache /data1/saved_caches/system-hints-KeyCache-b.db
>>>> INFO [main] 2014-04-29 14:44:35,737 ColumnFamilyStore.java (line 248)
>>>> Initializing system.schema_keyspaces
>>>>
>>>>
>>>
>>>
>>> On Tue, Apr 29, 2014 at 6:07 PM, Yatong Zhang <bl...@gmail.com>
>>> wrote:
>>>
>>> I am pretty sure the disk has plenty of space, I am sure of that. I
>>>> restarted cassandra and everything went fine again.
>>>>
>>>> It's really wired
>>>>
>>>>
>>>> On Tue, Apr 29, 2014 at 5:58 PM, Sylvain Lebresne <sylvain@datastax.com
>>>> >wrote:
>>>>
>>>> The important part of that stack trace is "java.io.IOException: No
>>>>> space
>>>>> left on device", your disks are full (and it's not really a bug that
>>>>> Cassandra error out in that case).
>>>>>
>>>>> --
>>>>> Sylvain
>>>>>
>>>>>
>>>>> On Tue, Apr 29, 2014 at 11:09 AM, Yatong Zhang <bl...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi there,
>>>>>>
>>>>>> Sorry if this is not the right place to report bugs. I am using 2.0.7
>>>>>>
>>>>> and I
>>>>>
>>>>>> have a 10 boxes clusters with about 200TB capacity. I just found I
>>>>>> had 3
>>>>>> boxes with error exceptions. With datastax opscenter I can see these
>>>>>>
>>>>> three
>>>>>
>>>>>> nodes lost connections (no reponse), but after I sshed to these
>>>>>> server,
>>>>>> cassandara were still running, and the 'system.log' still had logs.
>>>>>>
>>>>>> I think this might be a bug so any one would kindly help to
>>>>>> investigate
>>>>>> into it? Thanks~
>>>>>>
>>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249
>>>>>>
>>>>> CassandraDaemon.java
>>>>>
>>>>>> (line 198) Exception in thread Thread[CompactionExecutor:1,1,main]
>>>>>>> FSWriteError in
>>>>>>>
>>>>>> /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db
>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(
>>>>> SSTableWriter.java:475)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>> org.apache.cassandra.io.util.FileUtils.closeQuietly(
>>>>> FileUtils.java:212)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.cassandra.io.sstable.SSTableWriter.abort(
>>>>> SSTableWriter.java:301)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.cassandra.db.compaction.CompactionTask.
>>>>> runWith(CompactionTask.java:209)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(
>>>>> DiskAwareRunnable.java:48)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>> org.apache.cassandra.utils.WrappedRunnable.run(
>>>>> WrappedRunnable.java:28)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(
>>>>> CompactionTask.java:60)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(
>>>>> AbstractCompactionTask.java:59)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.cassandra.db.compaction.CompactionManager$
>>>>> BackgroundCompactionTask.run(CompactionManager.java:197)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>> call(Executors.java:471)
>>>>>
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>> ThreadPoolExecutor.java:1145)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>> ThreadPoolExecutor.java:615)
>>>>>
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>> Caused by: java.io.IOException: No space left on device
>>>>>>> at java.io.FileOutputStream.write(Native Method)
>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:
>>>>>>> 295)
>>>>>>> at
>>>>>>>
>>>>>> java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.cassandra.utils.BloomFilterSerializer.serialize(
>>>>> BloomFilterSerializer.java:34)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.cassandra.utils.Murmur3BloomFilter$
>>>>> Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>> org.apache.cassandra.utils.FilterFactory.serialize(
>>>>> FilterFactory.java:41)
>>>>>
>>>>>> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(
>>>>> SSTableWriter.java:468)
>>>>>
>>>>>> ... 13 more
>>>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406
>>>>>>>
>>>>>> StorageService.java
>>>>>
>>>>>> (line 367) Stopping gossiper
>>>>>>> WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406
>>>>>>>
>>>>>> StorageService.java
>>>>>
>>>>>> (line 281) Stopping gossip by operator request
>>>>>>> INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java
>>>>>>>
>>>>>> (line
>>>>>
>>>>>> 1271) Announcing shutdown
>>>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406
>>>>>>>
>>>>>> StorageService.java
>>>>>
>>>>>> (line 372) Stopping RPC server
>>>>>>> INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406
>>>>>>> ThriftServer.java
>>>>>>> (line 141) Stop listening to thrift clients
>>>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417
>>>>>>>
>>>>>> StorageService.java
>>>>>
>>>>>> (line 377) Stopping native transport
>>>>>>> INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java
>>>>>>> (line
>>>>>>> 181) Stop listening for CQL clients
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: bugs report for 2.0.7
Posted by Yatong Zhang <bl...@gmail.com>.
Thanks for the response. I've checked the system logs and harddisk smartd
info, and no errors found. Any hints to locate the problem?
On Wed, Apr 30, 2014 at 9:26 AM, Michael Shuler <mi...@pbandjelly.org>wrote:
> Then you likely need to fix your I/O problem. The most recent error you
> posted is an EOFException - the file being read ended unexpectedly.
> Probably when you ran out of disk space.
>
> --
> Michael
>
>
> On 04/29/2014 07:48 PM, Yatong Zhang wrote:
>
>> Here is another type of exception, seems all are I/O related:
>>
>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,548 SSTableReader.java
>> (line
>>
>>> 223) Opening
>>> /data2/cass/system/compaction_history/system-compaction_history-jb-6956
>>> (447252 bytes)
>>> INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,553 SSTableReader.java
>>> (line 223) Opening
>>> /data2/cass/system/compaction_history/system-compaction_history-jb-6958
>>> (257 bytes)
>>> INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,554 SSTableReader.java
>>> (line 223) Opening
>>> /data2/cass/system/compaction_history/system-compaction_history-jb-6957
>>> (257 bytes)
>>> INFO [main] 2014-04-29 14:44:35,592 ColumnFamilyStore.java (line 248)
>>> Initializing system.batchlog
>>> INFO [main] 2014-04-29 14:44:35,596 ColumnFamilyStore.java (line 248)
>>> Initializing system.sstable_activity
>>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,601 SSTableReader.java
>>> (line 223) Opening
>>> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8084
>>> (1562
>>> bytes)
>>> INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,604 SSTableReader.java
>>> (line 223) Opening
>>> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8083
>>> (2075
>>> bytes)
>>> INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,605 SSTableReader.java
>>> (line 223) Opening
>>> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8085
>>> (1555
>>> bytes)
>>> INFO [main] 2014-04-29 14:44:35,687 AutoSavingCache.java (line 114)
>>> reading saved cache
>>> /data1/saved_caches/system-sstable_activity-KeyCache-b.db
>>> INFO [main] 2014-04-29 14:44:35,696 ColumnFamilyStore.java (line 248)
>>> Initializing system.peer_events
>>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,697 SSTableReader.java
>>> (line 223) Opening /data4/cass/system/peer_events/system-peer_events-jb-
>>> 181
>>> (12342 bytes)
>>> INFO [main] 2014-04-29 14:44:35,717 ColumnFamilyStore.java (line 248)
>>> Initializing system.compactions_in_progress
>>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,718 SSTableReader.java
>>> (line 223) Opening
>>> /data5/cass/system/compactions_in_progress/system-compactions_in_
>>> progress-jb-36448
>>> (167 bytes)
>>> ERROR [SSTableBatchOpen:1] 2014-04-29 14:44:35,730 CassandraDaemon.java
>>> (line 198) Exception in thread Thread[SSTableBatchOpen:1,5,main]
>>> org.apache.cassandra.io.sstable.CorruptSSTableException:
>>> java.io.EOFException
>>> at
>>> org.apache.cassandra.io.compress.CompressionMetadata.<
>>> init>(CompressionMetadata.java:110)
>>> at
>>> org.apache.cassandra.io.compress.CompressionMetadata.
>>> create(CompressionMetadata.java:64)
>>> at
>>> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile
>>> $Builder.complete(CompressedPoolingSegmentedFile.java:42)
>>> at
>>> org.apache.cassandra.io.sstable.SSTableReader.load(
>>> SSTableReader.java:458)
>>> at
>>> org.apache.cassandra.io.sstable.SSTableReader.load(
>>> SSTableReader.java:422)
>>> at
>>> org.apache.cassandra.io.sstable.SSTableReader.open(
>>> SSTableReader.java:203)
>>> at
>>> org.apache.cassandra.io.sstable.SSTableReader.open(
>>> SSTableReader.java:184)
>>> at
>>> org.apache.cassandra.io.sstable.SSTableReader$1.run(
>>> SSTableReader.java:264)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>> Caused by: java.io.EOFException
>>> at
>>> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
>>> at java.io.DataInputStream.readUTF(DataInputStream.java:589)
>>> at java.io.DataInputStream.readUTF(DataInputStream.java:564)
>>> at
>>> org.apache.cassandra.io.compress.CompressionMetadata.<
>>> init>(CompressionMetadata.java:85)
>>> ... 12 more
>>> INFO [main] 2014-04-29 14:44:35,733 ColumnFamilyStore.java (line 248)
>>> Initializing system.hints
>>> INFO [main] 2014-04-29 14:44:35,734 AutoSavingCache.java (line 114)
>>> reading saved cache /data1/saved_caches/system-hints-KeyCache-b.db
>>> INFO [main] 2014-04-29 14:44:35,737 ColumnFamilyStore.java (line 248)
>>> Initializing system.schema_keyspaces
>>>
>>>
>>
>>
>> On Tue, Apr 29, 2014 at 6:07 PM, Yatong Zhang <bl...@gmail.com>
>> wrote:
>>
>> I am pretty sure the disk has plenty of space, I am sure of that. I
>>> restarted cassandra and everything went fine again.
>>>
>>> It's really wired
>>>
>>>
>>> On Tue, Apr 29, 2014 at 5:58 PM, Sylvain Lebresne <sylvain@datastax.com
>>> >wrote:
>>>
>>> The important part of that stack trace is "java.io.IOException: No space
>>>> left on device", your disks are full (and it's not really a bug that
>>>> Cassandra error out in that case).
>>>>
>>>> --
>>>> Sylvain
>>>>
>>>>
>>>> On Tue, Apr 29, 2014 at 11:09 AM, Yatong Zhang <bl...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi there,
>>>>>
>>>>> Sorry if this is not the right place to report bugs. I am using 2.0.7
>>>>>
>>>> and I
>>>>
>>>>> have a 10 boxes clusters with about 200TB capacity. I just found I had
>>>>> 3
>>>>> boxes with error exceptions. With datastax opscenter I can see these
>>>>>
>>>> three
>>>>
>>>>> nodes lost connections (no reponse), but after I sshed to these server,
>>>>> cassandara were still running, and the 'system.log' still had logs.
>>>>>
>>>>> I think this might be a bug so any one would kindly help to investigate
>>>>> into it? Thanks~
>>>>>
>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249
>>>>>
>>>> CassandraDaemon.java
>>>>
>>>>> (line 198) Exception in thread Thread[CompactionExecutor:1,1,main]
>>>>>> FSWriteError in
>>>>>>
>>>>> /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db
>>>>>
>>>>>> at
>>>>>>
>>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(
>>>> SSTableWriter.java:475)
>>>>
>>>>> at
>>>>>>
>>>>>> org.apache.cassandra.io.util.FileUtils.closeQuietly(
>>>> FileUtils.java:212)
>>>>
>>>>> at
>>>>>>
>>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableWriter.abort(
>>>> SSTableWriter.java:301)
>>>>
>>>>> at
>>>>>>
>>>>>>
>>>>> org.apache.cassandra.db.compaction.CompactionTask.
>>>> runWith(CompactionTask.java:209)
>>>>
>>>>> at
>>>>>>
>>>>>>
>>>>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(
>>>> DiskAwareRunnable.java:48)
>>>>
>>>>> at
>>>>>>
>>>>>> org.apache.cassandra.utils.WrappedRunnable.run(
>>>> WrappedRunnable.java:28)
>>>>
>>>>> at
>>>>>>
>>>>>>
>>>>> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(
>>>> CompactionTask.java:60)
>>>>
>>>>> at
>>>>>>
>>>>>>
>>>>> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(
>>>> AbstractCompactionTask.java:59)
>>>>
>>>>> at
>>>>>>
>>>>>>
>>>>> org.apache.cassandra.db.compaction.CompactionManager$
>>>> BackgroundCompactionTask.run(CompactionManager.java:197)
>>>>
>>>>> at
>>>>>>
>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>> call(Executors.java:471)
>>>>
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>>>>>>
>>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>> ThreadPoolExecutor.java:1145)
>>>>
>>>>> at
>>>>>>
>>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>> ThreadPoolExecutor.java:615)
>>>>
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> Caused by: java.io.IOException: No space left on device
>>>>>> at java.io.FileOutputStream.write(Native Method)
>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:295)
>>>>>> at
>>>>>>
>>>>> java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
>>>>
>>>>> at
>>>>>>
>>>>>>
>>>>> org.apache.cassandra.utils.BloomFilterSerializer.serialize(
>>>> BloomFilterSerializer.java:34)
>>>>
>>>>> at
>>>>>>
>>>>>>
>>>>> org.apache.cassandra.utils.Murmur3BloomFilter$
>>>> Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)
>>>>
>>>>> at
>>>>>>
>>>>>> org.apache.cassandra.utils.FilterFactory.serialize(
>>>> FilterFactory.java:41)
>>>>
>>>>> at
>>>>>>
>>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(
>>>> SSTableWriter.java:468)
>>>>
>>>>> ... 13 more
>>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406
>>>>>>
>>>>> StorageService.java
>>>>
>>>>> (line 367) Stopping gossiper
>>>>>> WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406
>>>>>>
>>>>> StorageService.java
>>>>
>>>>> (line 281) Stopping gossip by operator request
>>>>>> INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java
>>>>>>
>>>>> (line
>>>>
>>>>> 1271) Announcing shutdown
>>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406
>>>>>>
>>>>> StorageService.java
>>>>
>>>>> (line 372) Stopping RPC server
>>>>>> INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406
>>>>>> ThriftServer.java
>>>>>> (line 141) Stop listening to thrift clients
>>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417
>>>>>>
>>>>> StorageService.java
>>>>
>>>>> (line 377) Stopping native transport
>>>>>> INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java
>>>>>> (line
>>>>>> 181) Stop listening for CQL clients
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
Re: bugs report for 2.0.7
Posted by Michael Shuler <mi...@pbandjelly.org>.
Then you likely need to fix your I/O problem. The most recent error you
posted is an EOFException - the file being read ended unexpectedly.
Probably when you ran out of disk space.
--
Michael
On 04/29/2014 07:48 PM, Yatong Zhang wrote:
> Here is another type of exception, seems all are I/O related:
>
> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,548 SSTableReader.java (line
>> 223) Opening
>> /data2/cass/system/compaction_history/system-compaction_history-jb-6956
>> (447252 bytes)
>> INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,553 SSTableReader.java
>> (line 223) Opening
>> /data2/cass/system/compaction_history/system-compaction_history-jb-6958
>> (257 bytes)
>> INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,554 SSTableReader.java
>> (line 223) Opening
>> /data2/cass/system/compaction_history/system-compaction_history-jb-6957
>> (257 bytes)
>> INFO [main] 2014-04-29 14:44:35,592 ColumnFamilyStore.java (line 248)
>> Initializing system.batchlog
>> INFO [main] 2014-04-29 14:44:35,596 ColumnFamilyStore.java (line 248)
>> Initializing system.sstable_activity
>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,601 SSTableReader.java
>> (line 223) Opening
>> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8084 (1562
>> bytes)
>> INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,604 SSTableReader.java
>> (line 223) Opening
>> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8083 (2075
>> bytes)
>> INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,605 SSTableReader.java
>> (line 223) Opening
>> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8085 (1555
>> bytes)
>> INFO [main] 2014-04-29 14:44:35,687 AutoSavingCache.java (line 114)
>> reading saved cache
>> /data1/saved_caches/system-sstable_activity-KeyCache-b.db
>> INFO [main] 2014-04-29 14:44:35,696 ColumnFamilyStore.java (line 248)
>> Initializing system.peer_events
>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,697 SSTableReader.java
>> (line 223) Opening /data4/cass/system/peer_events/system-peer_events-jb-181
>> (12342 bytes)
>> INFO [main] 2014-04-29 14:44:35,717 ColumnFamilyStore.java (line 248)
>> Initializing system.compactions_in_progress
>> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,718 SSTableReader.java
>> (line 223) Opening
>> /data5/cass/system/compactions_in_progress/system-compactions_in_progress-jb-36448
>> (167 bytes)
>> ERROR [SSTableBatchOpen:1] 2014-04-29 14:44:35,730 CassandraDaemon.java
>> (line 198) Exception in thread Thread[SSTableBatchOpen:1,5,main]
>> org.apache.cassandra.io.sstable.CorruptSSTableException:
>> java.io.EOFException
>> at
>> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:110)
>> at
>> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64)
>> at
>> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
>> at
>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:458)
>> at
>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:422)
>> at
>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:203)
>> at
>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:184)
>> at
>> org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:264)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> Caused by: java.io.EOFException
>> at
>> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
>> at java.io.DataInputStream.readUTF(DataInputStream.java:589)
>> at java.io.DataInputStream.readUTF(DataInputStream.java:564)
>> at
>> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:85)
>> ... 12 more
>> INFO [main] 2014-04-29 14:44:35,733 ColumnFamilyStore.java (line 248)
>> Initializing system.hints
>> INFO [main] 2014-04-29 14:44:35,734 AutoSavingCache.java (line 114)
>> reading saved cache /data1/saved_caches/system-hints-KeyCache-b.db
>> INFO [main] 2014-04-29 14:44:35,737 ColumnFamilyStore.java (line 248)
>> Initializing system.schema_keyspaces
>>
>
>
>
> On Tue, Apr 29, 2014 at 6:07 PM, Yatong Zhang <bl...@gmail.com> wrote:
>
>> I am pretty sure the disk has plenty of space, I am sure of that. I
>> restarted cassandra and everything went fine again.
>>
>> It's really wired
>>
>>
>> On Tue, Apr 29, 2014 at 5:58 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>
>>> The important part of that stack trace is "java.io.IOException: No space
>>> left on device", your disks are full (and it's not really a bug that
>>> Cassandra error out in that case).
>>>
>>> --
>>> Sylvain
>>>
>>>
>>> On Tue, Apr 29, 2014 at 11:09 AM, Yatong Zhang <bl...@gmail.com>
>>> wrote:
>>>
>>>> Hi there,
>>>>
>>>> Sorry if this is not the right place to report bugs. I am using 2.0.7
>>> and I
>>>> have a 10 boxes clusters with about 200TB capacity. I just found I had 3
>>>> boxes with error exceptions. With datastax opscenter I can see these
>>> three
>>>> nodes lost connections (no reponse), but after I sshed to these server,
>>>> cassandara were still running, and the 'system.log' still had logs.
>>>>
>>>> I think this might be a bug so any one would kindly help to investigate
>>>> into it? Thanks~
>>>>
>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249
>>> CassandraDaemon.java
>>>>> (line 198) Exception in thread Thread[CompactionExecutor:1,1,main]
>>>>> FSWriteError in
>>>> /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db
>>>>> at
>>>>>
>>>>
>>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475)
>>>>> at
>>>>>
>>> org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)
>>>>> at
>>>>>
>>>>
>>> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301)
>>>>> at
>>>>>
>>>>
>>> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209)
>>>>> at
>>>>>
>>>>
>>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>>>>> at
>>>>>
>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>>>>> at
>>>>>
>>>>
>>> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>>>>> at
>>>>>
>>>>
>>> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>>>>> at
>>>>>
>>>>
>>> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
>>>>> at
>>>>>
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at
>>>>>
>>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>>
>>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> Caused by: java.io.IOException: No space left on device
>>>>> at java.io.FileOutputStream.write(Native Method)
>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:295)
>>>>> at
>>> java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
>>>>> at
>>>>>
>>>>
>>> org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34)
>>>>> at
>>>>>
>>>>
>>> org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)
>>>>> at
>>>>>
>>> org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41)
>>>>> at
>>>>>
>>>>
>>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:468)
>>>>> ... 13 more
>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406
>>> StorageService.java
>>>>> (line 367) Stopping gossiper
>>>>> WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406
>>> StorageService.java
>>>>> (line 281) Stopping gossip by operator request
>>>>> INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java
>>> (line
>>>>> 1271) Announcing shutdown
>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406
>>> StorageService.java
>>>>> (line 372) Stopping RPC server
>>>>> INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406 ThriftServer.java
>>>>> (line 141) Stop listening to thrift clients
>>>>> ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417
>>> StorageService.java
>>>>> (line 377) Stopping native transport
>>>>> INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java (line
>>>>> 181) Stop listening for CQL clients
>>>>>
>>>>
>>>
>>
>>
>
Re: bugs report for 2.0.7
Posted by Yatong Zhang <bl...@gmail.com>.
Here is another type of exception, seems all are I/O related:
INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,548 SSTableReader.java (line
> 223) Opening
> /data2/cass/system/compaction_history/system-compaction_history-jb-6956
> (447252 bytes)
> INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,553 SSTableReader.java
> (line 223) Opening
> /data2/cass/system/compaction_history/system-compaction_history-jb-6958
> (257 bytes)
> INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,554 SSTableReader.java
> (line 223) Opening
> /data2/cass/system/compaction_history/system-compaction_history-jb-6957
> (257 bytes)
> INFO [main] 2014-04-29 14:44:35,592 ColumnFamilyStore.java (line 248)
> Initializing system.batchlog
> INFO [main] 2014-04-29 14:44:35,596 ColumnFamilyStore.java (line 248)
> Initializing system.sstable_activity
> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,601 SSTableReader.java
> (line 223) Opening
> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8084 (1562
> bytes)
> INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,604 SSTableReader.java
> (line 223) Opening
> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8083 (2075
> bytes)
> INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,605 SSTableReader.java
> (line 223) Opening
> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8085 (1555
> bytes)
> INFO [main] 2014-04-29 14:44:35,687 AutoSavingCache.java (line 114)
> reading saved cache
> /data1/saved_caches/system-sstable_activity-KeyCache-b.db
> INFO [main] 2014-04-29 14:44:35,696 ColumnFamilyStore.java (line 248)
> Initializing system.peer_events
> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,697 SSTableReader.java
> (line 223) Opening /data4/cass/system/peer_events/system-peer_events-jb-181
> (12342 bytes)
> INFO [main] 2014-04-29 14:44:35,717 ColumnFamilyStore.java (line 248)
> Initializing system.compactions_in_progress
> INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,718 SSTableReader.java
> (line 223) Opening
> /data5/cass/system/compactions_in_progress/system-compactions_in_progress-jb-36448
> (167 bytes)
> ERROR [SSTableBatchOpen:1] 2014-04-29 14:44:35,730 CassandraDaemon.java
> (line 198) Exception in thread Thread[SSTableBatchOpen:1,5,main]
> org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.EOFException
> at
> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:110)
> at
> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64)
> at
> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
> at
> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:458)
> at
> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:422)
> at
> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:203)
> at
> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:184)
> at
> org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:264)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.EOFException
> at
> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
> at java.io.DataInputStream.readUTF(DataInputStream.java:589)
> at java.io.DataInputStream.readUTF(DataInputStream.java:564)
> at
> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:85)
> ... 12 more
> INFO [main] 2014-04-29 14:44:35,733 ColumnFamilyStore.java (line 248)
> Initializing system.hints
> INFO [main] 2014-04-29 14:44:35,734 AutoSavingCache.java (line 114)
> reading saved cache /data1/saved_caches/system-hints-KeyCache-b.db
> INFO [main] 2014-04-29 14:44:35,737 ColumnFamilyStore.java (line 248)
> Initializing system.schema_keyspaces
>
On Tue, Apr 29, 2014 at 6:07 PM, Yatong Zhang <bl...@gmail.com> wrote:
> I am pretty sure the disk has plenty of space, I am sure of that. I
> restarted cassandra and everything went fine again.
>
> It's really wired
>
>
> On Tue, Apr 29, 2014 at 5:58 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>> The important part of that stack trace is "java.io.IOException: No space
>> left on device", your disks are full (and it's not really a bug that
>> Cassandra error out in that case).
>>
>> --
>> Sylvain
>>
>>
>> On Tue, Apr 29, 2014 at 11:09 AM, Yatong Zhang <bl...@gmail.com>
>> wrote:
>>
>> > Hi there,
>> >
>> > Sorry if this is not the right place to report bugs. I am using 2.0.7
>> and I
>> > have a 10 boxes clusters with about 200TB capacity. I just found I had 3
>> > boxes with error exceptions. With datastax opscenter I can see these
>> three
>> > nodes lost connections (no reponse), but after I sshed to these server,
>> > cassandara were still running, and the 'system.log' still had logs.
>> >
>> > I think this might be a bug so any one would kindly help to investigate
>> > into it? Thanks~
>> >
>> > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249
>> CassandraDaemon.java
>> > > (line 198) Exception in thread Thread[CompactionExecutor:1,1,main]
>> > > FSWriteError in
>> > /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db
>> > > at
>> > >
>> >
>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475)
>> > > at
>> > >
>> org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)
>> > > at
>> > >
>> >
>> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301)
>> > > at
>> > >
>> >
>> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209)
>> > > at
>> > >
>> >
>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>> > > at
>> > >
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>> > > at
>> > >
>> >
>> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>> > > at
>> > >
>> >
>> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>> > > at
>> > >
>> >
>> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
>> > > at
>> > >
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> > > at
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > > at
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > > at java.lang.Thread.run(Thread.java:744)
>> > > Caused by: java.io.IOException: No space left on device
>> > > at java.io.FileOutputStream.write(Native Method)
>> > > at java.io.FileOutputStream.write(FileOutputStream.java:295)
>> > > at
>> java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
>> > > at
>> > >
>> >
>> org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34)
>> > > at
>> > >
>> >
>> org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)
>> > > at
>> > >
>> org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41)
>> > > at
>> > >
>> >
>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:468)
>> > > ... 13 more
>> > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406
>> StorageService.java
>> > > (line 367) Stopping gossiper
>> > > WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406
>> StorageService.java
>> > > (line 281) Stopping gossip by operator request
>> > > INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java
>> (line
>> > > 1271) Announcing shutdown
>> > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406
>> StorageService.java
>> > > (line 372) Stopping RPC server
>> > > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406 ThriftServer.java
>> > > (line 141) Stop listening to thrift clients
>> > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417
>> StorageService.java
>> > > (line 377) Stopping native transport
>> > > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java (line
>> > > 181) Stop listening for CQL clients
>> > >
>> >
>>
>
>
Re: bugs report for 2.0.7
Posted by Yatong Zhang <bl...@gmail.com>.
I am pretty sure the disk has plenty of space, I am sure of that. I
restarted cassandra and everything went fine again.
It's really wired
On Tue, Apr 29, 2014 at 5:58 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
> The important part of that stack trace is "java.io.IOException: No space
> left on device", your disks are full (and it's not really a bug that
> Cassandra error out in that case).
>
> --
> Sylvain
>
>
> On Tue, Apr 29, 2014 at 11:09 AM, Yatong Zhang <bl...@gmail.com>
> wrote:
>
> > Hi there,
> >
> > Sorry if this is not the right place to report bugs. I am using 2.0.7
> and I
> > have a 10 boxes clusters with about 200TB capacity. I just found I had 3
> > boxes with error exceptions. With datastax opscenter I can see these
> three
> > nodes lost connections (no reponse), but after I sshed to these server,
> > cassandara were still running, and the 'system.log' still had logs.
> >
> > I think this might be a bug so any one would kindly help to investigate
> > into it? Thanks~
> >
> > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249 CassandraDaemon.java
> > > (line 198) Exception in thread Thread[CompactionExecutor:1,1,main]
> > > FSWriteError in
> > /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db
> > > at
> > >
> >
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475)
> > > at
> > > org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)
> > > at
> > >
> >
> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301)
> > > at
> > >
> >
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209)
> > > at
> > >
> >
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> > > at
> > > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> > > at
> > >
> >
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> > > at
> > >
> >
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> > > at
> > >
> >
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:744)
> > > Caused by: java.io.IOException: No space left on device
> > > at java.io.FileOutputStream.write(Native Method)
> > > at java.io.FileOutputStream.write(FileOutputStream.java:295)
> > > at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
> > > at
> > >
> >
> org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34)
> > > at
> > >
> >
> org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)
> > > at
> > >
> org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41)
> > > at
> > >
> >
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:468)
> > > ... 13 more
> > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406
> StorageService.java
> > > (line 367) Stopping gossiper
> > > WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406
> StorageService.java
> > > (line 281) Stopping gossip by operator request
> > > INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java
> (line
> > > 1271) Announcing shutdown
> > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406
> StorageService.java
> > > (line 372) Stopping RPC server
> > > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406 ThriftServer.java
> > > (line 141) Stop listening to thrift clients
> > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417
> StorageService.java
> > > (line 377) Stopping native transport
> > > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java (line
> > > 181) Stop listening for CQL clients
> > >
> >
>
Re: bugs report for 2.0.7
Posted by Sylvain Lebresne <sy...@datastax.com>.
The important part of that stack trace is "java.io.IOException: No space
left on device", your disks are full (and it's not really a bug that
Cassandra error out in that case).
--
Sylvain
On Tue, Apr 29, 2014 at 11:09 AM, Yatong Zhang <bl...@gmail.com> wrote:
> Hi there,
>
> Sorry if this is not the right place to report bugs. I am using 2.0.7 and I
> have a 10 boxes clusters with about 200TB capacity. I just found I had 3
> boxes with error exceptions. With datastax opscenter I can see these three
> nodes lost connections (no reponse), but after I sshed to these server,
> cassandara were still running, and the 'system.log' still had logs.
>
> I think this might be a bug so any one would kindly help to investigate
> into it? Thanks~
>
> ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249 CassandraDaemon.java
> > (line 198) Exception in thread Thread[CompactionExecutor:1,1,main]
> > FSWriteError in
> /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db
> > at
> >
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475)
> > at
> > org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)
> > at
> >
> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301)
> > at
> >
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209)
> > at
> >
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> > at
> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> > at
> >
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> > at
> >
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> > at
> >
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:744)
> > Caused by: java.io.IOException: No space left on device
> > at java.io.FileOutputStream.write(Native Method)
> > at java.io.FileOutputStream.write(FileOutputStream.java:295)
> > at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
> > at
> >
> org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34)
> > at
> >
> org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)
> > at
> > org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41)
> > at
> >
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:468)
> > ... 13 more
> > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406 StorageService.java
> > (line 367) Stopping gossiper
> > WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406 StorageService.java
> > (line 281) Stopping gossip by operator request
> > INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java (line
> > 1271) Announcing shutdown
> > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406 StorageService.java
> > (line 372) Stopping RPC server
> > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406 ThriftServer.java
> > (line 141) Stop listening to thrift clients
> > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417 StorageService.java
> > (line 377) Stopping native transport
> > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java (line
> > 181) Stop listening for CQL clients
> >
>