You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by ruslan usifov <ru...@gmail.com> on 2012/05/17 11:57:49 UTC
Exception when truncate
Hello
I have follow situation on our test server:
from cassandra-cli i try to use
truncate purchase_history;
3 times i got:
[default@township_6waves] truncate purchase_history;
null
UnavailableException()
at org.apache.cassandra.thrift.Cassandra$truncate_result.read(Cassandra.java:20212)
at org.apache.cassandra.thrift.Cassandra$Client.recv_truncate(Cassandra.java:1077)
at org.apache.cassandra.thrift.Cassandra$Client.truncate(Cassandra.java:1052)
at org.apache.cassandra.cli.CliClient.executeTruncate(CliClient.java:1445)
at org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:272)
at org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:220)
at org.apache.cassandra.cli.CliMain.main(CliMain.java:348)
So this looks that truncate goes very slow and too long, than
rpc_timeout_in_ms: 10000 (this can happens because we have very slow
disck on test machine)
But in in cassandra system log i see follow exception:
ERROR [MutationStage:7022] 2012-05-17 12:19:14,356
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[MutationStage:7022,5,main]
java.io.IOError: java.io.IOException: unable to mkdirs
/home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-purchase_history
at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1433)
at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1462)
at org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.java:1657)
at org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandler.java:50)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: unable to mkdirs
/home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-purchase_history
at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:140)
at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:131)
at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1409)
... 7 more
Also i see that in snapshort dir already exists
1337242754356-purchase_history directory, so i think that snapshort
names that generate cassandra not uniquely.
PS: We use cassandra 1.0.10 on Ubuntu 10.0.4-LTS
Re: Exception when truncate
Posted by ruslan usifov <ru...@gmail.com>.
It's look s very strange but yes. Now i can't reproduce this
2012/5/22 aaron morton <aa...@thelastpickle.com>:
> The first part of the name is the current system time in milliseconds.
>
> If you run it twice do you get log messages about failing to create the same
> directory twice ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 21/05/2012, at 5:09 AM, ruslan usifov wrote:
>
> I think as you, but this is not true, there are not any permissions
> issue. And as i said before, cassandra try to create directory for
> snapshort that already exists
>
> 2012/5/19 Jonathan Ellis <jb...@gmail.com>:
>
> Sounds like you have a permissions problem. Cassandra creates a
>
> subdirectory for each snapshot.
>
>
> On Thu, May 17, 2012 at 4:57 AM, ruslan usifov <ru...@gmail.com>
> wrote:
>
> Hello
>
>
> I have follow situation on our test server:
>
>
> from cassandra-cli i try to use
>
>
> truncate purchase_history;
>
>
> 3 times i got:
>
>
> [default@township_6waves] truncate purchase_history;
>
> null
>
> UnavailableException()
>
> at
> org.apache.cassandra.thrift.Cassandra$truncate_result.read(Cassandra.java:20212)
>
> at
> org.apache.cassandra.thrift.Cassandra$Client.recv_truncate(Cassandra.java:1077)
>
> at
> org.apache.cassandra.thrift.Cassandra$Client.truncate(Cassandra.java:1052)
>
> at
> org.apache.cassandra.cli.CliClient.executeTruncate(CliClient.java:1445)
>
> at
> org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:272)
>
> at
> org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:220)
>
> at org.apache.cassandra.cli.CliMain.main(CliMain.java:348)
>
>
>
> So this looks that truncate goes very slow and too long, than
>
> rpc_timeout_in_ms: 10000 (this can happens because we have very slow
>
> disck on test machine)
>
>
> But in in cassandra system log i see follow exception:
>
>
>
> ERROR [MutationStage:7022] 2012-05-17 12:19:14,356
>
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>
> Thread[MutationStage:7022,5,main]
>
> java.io.IOError: java.io.IOException: unable to mkdirs
>
> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-purchase_history
>
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1433)
>
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1462)
>
> at
> org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.java:1657)
>
> at
> org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandler.java:50)
>
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: unable to mkdirs
>
> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-purchase_history
>
> at
> org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:140)
>
> at
> org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:131)
>
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1409)
>
> ... 7 more
>
>
>
> Also i see that in snapshort dir already exists
>
> 1337242754356-purchase_history directory, so i think that snapshort
>
> names that generate cassandra not uniquely.
>
>
> PS: We use cassandra 1.0.10 on Ubuntu 10.0.4-LTS
>
>
>
>
> --
>
> Jonathan Ellis
>
> Project Chair, Apache Cassandra
>
> co-founder of DataStax, the source for professional Cassandra support
>
> http://www.datastax.com
>
>
Re: Exception when truncate
Posted by aaron morton <aa...@thelastpickle.com>.
The first part of the name is the current system time in milliseconds.
If you run it twice do you get log messages about failing to create the same directory twice ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 21/05/2012, at 5:09 AM, ruslan usifov wrote:
> I think as you, but this is not true, there are not any permissions
> issue. And as i said before, cassandra try to create directory for
> snapshort that already exists
>
> 2012/5/19 Jonathan Ellis <jb...@gmail.com>:
>> Sounds like you have a permissions problem. Cassandra creates a
>> subdirectory for each snapshot.
>>
>> On Thu, May 17, 2012 at 4:57 AM, ruslan usifov <ru...@gmail.com> wrote:
>>> Hello
>>>
>>> I have follow situation on our test server:
>>>
>>> from cassandra-cli i try to use
>>>
>>> truncate purchase_history;
>>>
>>> 3 times i got:
>>>
>>> [default@township_6waves] truncate purchase_history;
>>> null
>>> UnavailableException()
>>> at org.apache.cassandra.thrift.Cassandra$truncate_result.read(Cassandra.java:20212)
>>> at org.apache.cassandra.thrift.Cassandra$Client.recv_truncate(Cassandra.java:1077)
>>> at org.apache.cassandra.thrift.Cassandra$Client.truncate(Cassandra.java:1052)
>>> at org.apache.cassandra.cli.CliClient.executeTruncate(CliClient.java:1445)
>>> at org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:272)
>>> at org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:220)
>>> at org.apache.cassandra.cli.CliMain.main(CliMain.java:348)
>>>
>>>
>>> So this looks that truncate goes very slow and too long, than
>>> rpc_timeout_in_ms: 10000 (this can happens because we have very slow
>>> disck on test machine)
>>>
>>> But in in cassandra system log i see follow exception:
>>>
>>>
>>> ERROR [MutationStage:7022] 2012-05-17 12:19:14,356
>>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>>> Thread[MutationStage:7022,5,main]
>>> java.io.IOError: java.io.IOException: unable to mkdirs
>>> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-purchase_history
>>> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1433)
>>> at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1462)
>>> at org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.java:1657)
>>> at org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandler.java:50)
>>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.io.IOException: unable to mkdirs
>>> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-purchase_history
>>> at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:140)
>>> at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:131)
>>> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1409)
>>> ... 7 more
>>>
>>>
>>> Also i see that in snapshort dir already exists
>>> 1337242754356-purchase_history directory, so i think that snapshort
>>> names that generate cassandra not uniquely.
>>>
>>> PS: We use cassandra 1.0.10 on Ubuntu 10.0.4-LTS
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
Re: Exception when truncate
Posted by ruslan usifov <ru...@gmail.com>.
I think as you, but this is not true, there are not any permissions
issue. And as i said before, cassandra try to create directory for
snapshort that already exists
2012/5/19 Jonathan Ellis <jb...@gmail.com>:
> Sounds like you have a permissions problem. Cassandra creates a
> subdirectory for each snapshot.
>
> On Thu, May 17, 2012 at 4:57 AM, ruslan usifov <ru...@gmail.com> wrote:
>> Hello
>>
>> I have follow situation on our test server:
>>
>> from cassandra-cli i try to use
>>
>> truncate purchase_history;
>>
>> 3 times i got:
>>
>> [default@township_6waves] truncate purchase_history;
>> null
>> UnavailableException()
>> at org.apache.cassandra.thrift.Cassandra$truncate_result.read(Cassandra.java:20212)
>> at org.apache.cassandra.thrift.Cassandra$Client.recv_truncate(Cassandra.java:1077)
>> at org.apache.cassandra.thrift.Cassandra$Client.truncate(Cassandra.java:1052)
>> at org.apache.cassandra.cli.CliClient.executeTruncate(CliClient.java:1445)
>> at org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:272)
>> at org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:220)
>> at org.apache.cassandra.cli.CliMain.main(CliMain.java:348)
>>
>>
>> So this looks that truncate goes very slow and too long, than
>> rpc_timeout_in_ms: 10000 (this can happens because we have very slow
>> disck on test machine)
>>
>> But in in cassandra system log i see follow exception:
>>
>>
>> ERROR [MutationStage:7022] 2012-05-17 12:19:14,356
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[MutationStage:7022,5,main]
>> java.io.IOError: java.io.IOException: unable to mkdirs
>> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-purchase_history
>> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1433)
>> at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1462)
>> at org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.java:1657)
>> at org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandler.java:50)
>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.IOException: unable to mkdirs
>> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-purchase_history
>> at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:140)
>> at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:131)
>> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1409)
>> ... 7 more
>>
>>
>> Also i see that in snapshort dir already exists
>> 1337242754356-purchase_history directory, so i think that snapshort
>> names that generate cassandra not uniquely.
>>
>> PS: We use cassandra 1.0.10 on Ubuntu 10.0.4-LTS
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: Exception when truncate
Posted by Jonathan Ellis <jb...@gmail.com>.
Sounds like you have a permissions problem. Cassandra creates a
subdirectory for each snapshot.
On Thu, May 17, 2012 at 4:57 AM, ruslan usifov <ru...@gmail.com> wrote:
> Hello
>
> I have follow situation on our test server:
>
> from cassandra-cli i try to use
>
> truncate purchase_history;
>
> 3 times i got:
>
> [default@township_6waves] truncate purchase_history;
> null
> UnavailableException()
> at org.apache.cassandra.thrift.Cassandra$truncate_result.read(Cassandra.java:20212)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_truncate(Cassandra.java:1077)
> at org.apache.cassandra.thrift.Cassandra$Client.truncate(Cassandra.java:1052)
> at org.apache.cassandra.cli.CliClient.executeTruncate(CliClient.java:1445)
> at org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:272)
> at org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:220)
> at org.apache.cassandra.cli.CliMain.main(CliMain.java:348)
>
>
> So this looks that truncate goes very slow and too long, than
> rpc_timeout_in_ms: 10000 (this can happens because we have very slow
> disck on test machine)
>
> But in in cassandra system log i see follow exception:
>
>
> ERROR [MutationStage:7022] 2012-05-17 12:19:14,356
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[MutationStage:7022,5,main]
> java.io.IOError: java.io.IOException: unable to mkdirs
> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-purchase_history
> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1433)
> at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1462)
> at org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.java:1657)
> at org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandler.java:50)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: unable to mkdirs
> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-purchase_history
> at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:140)
> at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:131)
> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1409)
> ... 7 more
>
>
> Also i see that in snapshort dir already exists
> 1337242754356-purchase_history directory, so i think that snapshort
> names that generate cassandra not uniquely.
>
> PS: We use cassandra 1.0.10 on Ubuntu 10.0.4-LTS
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Exception when truncate
Posted by ruslan usifov <ru...@gmail.com>.
Its our test machine with one node in cluster:-)
2012/5/17 Jeremy Hanna <je...@gmail.com>:
> when doing a truncate, it has to talk to all of the nodes in the ring to perform the operation. by the error, it looks like one of the nodes was unreachable for some reason. you might do a nodetool ring in the cli do a 'describe cluster;' and see if your ring is okay.
>
> So I think the operation is just as fast, it just looks like it times out (20 seconds or something) when trying to perform the command against all of the nodes in the cluster.
>
> On May 17, 2012, at 9:36 AM, ruslan usifov wrote:
>
>> Maybe, something changes in cassandra 1.0.x for truncate mechanism,
>> because in cassandra 0.8 truncate executes much faster on the same
>> data
>>
>> 2012/5/17 Viktor Jevdokimov <Vi...@adform.com>:
>>> Truncate flushes all memtables to free up commit logs, and that on all nodes. So this takes time. Discussed on this list not so long ago.
>>>
>>> Watch for:
>>> https://issues.apache.org/jira/browse/CASSANDRA-3651
>>> https://issues.apache.org/jira/browse/CASSANDRA-4006
>>>
>>>
>>>
>>> Best regards / Pagarbiai
>>>
>>> Viktor Jevdokimov
>>> Senior Developer
>>>
>>> Email: Viktor.Jevdokimov@adform.com
>>> Phone: +370 5 212 3063
>>> Fax: +370 5 261 0453
>>>
>>> J. Jasinskio 16C,
>>> LT-01112 Vilnius,
>>> Lithuania
>>>
>>>
>>>
>>> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.> -----Original Message-----
>>>> From: ruslan usifov [mailto:ruslan.usifov@gmail.com]
>>>> Sent: Thursday, May 17, 2012 13:06
>>>> To: user@cassandra.apache.org
>>>> Subject: Re: Exception when truncate
>>>>
>>>> Also i miss understand why on empty CF(no any SStable) truncate heavy
>>>> loads disk??
>>>>
>>>> 2012/5/17 ruslan usifov <ru...@gmail.com>:
>>>>> Hello
>>>>>
>>>>> I have follow situation on our test server:
>>>>>
>>>>> from cassandra-cli i try to use
>>>>>
>>>>> truncate purchase_history;
>>>>>
>>>>> 3 times i got:
>>>>>
>>>>> [default@township_6waves] truncate purchase_history; null
>>>>> UnavailableException()
>>>>> at
>>>>> org.apache.cassandra.thrift.Cassandra$truncate_result.read(Cassandra.j
>>>>> ava:20212)
>>>>> at
>>>>> org.apache.cassandra.thrift.Cassandra$Client.recv_truncate(Cassandra.j
>>>>> ava:1077)
>>>>> at
>>>>> org.apache.cassandra.thrift.Cassandra$Client.truncate(Cassandra.java:1
>>>>> 052)
>>>>> at
>>>>> org.apache.cassandra.cli.CliClient.executeTruncate(CliClient.java:1445
>>>>> )
>>>>> at
>>>>> org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:
>>>>> 272)
>>>>> at
>>>>> org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.j
>>>>> ava:220)
>>>>> at org.apache.cassandra.cli.CliMain.main(CliMain.java:348)
>>>>>
>>>>>
>>>>> So this looks that truncate goes very slow and too long, than
>>>>> rpc_timeout_in_ms: 10000 (this can happens because we have very slow
>>>>> disck on test machine)
>>>>>
>>>>> But in in cassandra system log i see follow exception:
>>>>>
>>>>>
>>>>> ERROR [MutationStage:7022] 2012-05-17 12:19:14,356
>>>>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>>>>> Thread[MutationStage:7022,5,main]
>>>>> java.io.IOError: java.io.IOException: unable to mkdirs
>>>>>
>>>> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-
>>>> pur
>>>>> chase_history
>>>>> at
>>>>>
>>>> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(Column
>>>> F
>>>>> amilyStore.java:1433)
>>>>> at
>>>>>
>>>> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.j
>>>>> ava:1462)
>>>>> at
>>>>>
>>>> org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.j
>>>>> ava:1657)
>>>>> at
>>>>>
>>>> org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandle
>>>> r
>>>>> .java:50)
>>>>> at
>>>>>
>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.j
>>>>> ava:59)
>>>>> at
>>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu
>>>>> tor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>>>> java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.io.IOException: unable to mkdirs
>>>>>
>>>> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-
>>>> pur
>>>>> chase_history
>>>>> at
>>>>> org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:
>>>>> 140)
>>>>> at
>>>>> org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:
>>>>> 131)
>>>>> at
>>>>>
>>>> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(Column
>>>> F
>>>>> amilyStore.java:1409)
>>>>> ... 7 more
>>>>>
>>>>>
>>>>> Also i see that in snapshort dir already exists
>>>>> 1337242754356-purchase_history directory, so i think that snapshort
>>>>> names that generate cassandra not uniquely.
>>>>>
>>>>> PS: We use cassandra 1.0.10 on Ubuntu 10.0.4-LTS
>
Re: Exception when truncate
Posted by Jeremy Hanna <je...@gmail.com>.
when doing a truncate, it has to talk to all of the nodes in the ring to perform the operation. by the error, it looks like one of the nodes was unreachable for some reason. you might do a nodetool ring in the cli do a 'describe cluster;' and see if your ring is okay.
So I think the operation is just as fast, it just looks like it times out (20 seconds or something) when trying to perform the command against all of the nodes in the cluster.
On May 17, 2012, at 9:36 AM, ruslan usifov wrote:
> Maybe, something changes in cassandra 1.0.x for truncate mechanism,
> because in cassandra 0.8 truncate executes much faster on the same
> data
>
> 2012/5/17 Viktor Jevdokimov <Vi...@adform.com>:
>> Truncate flushes all memtables to free up commit logs, and that on all nodes. So this takes time. Discussed on this list not so long ago.
>>
>> Watch for:
>> https://issues.apache.org/jira/browse/CASSANDRA-3651
>> https://issues.apache.org/jira/browse/CASSANDRA-4006
>>
>>
>>
>> Best regards / Pagarbiai
>>
>> Viktor Jevdokimov
>> Senior Developer
>>
>> Email: Viktor.Jevdokimov@adform.com
>> Phone: +370 5 212 3063
>> Fax: +370 5 261 0453
>>
>> J. Jasinskio 16C,
>> LT-01112 Vilnius,
>> Lithuania
>>
>>
>>
>> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.> -----Original Message-----
>>> From: ruslan usifov [mailto:ruslan.usifov@gmail.com]
>>> Sent: Thursday, May 17, 2012 13:06
>>> To: user@cassandra.apache.org
>>> Subject: Re: Exception when truncate
>>>
>>> Also i miss understand why on empty CF(no any SStable) truncate heavy
>>> loads disk??
>>>
>>> 2012/5/17 ruslan usifov <ru...@gmail.com>:
>>>> Hello
>>>>
>>>> I have follow situation on our test server:
>>>>
>>>> from cassandra-cli i try to use
>>>>
>>>> truncate purchase_history;
>>>>
>>>> 3 times i got:
>>>>
>>>> [default@township_6waves] truncate purchase_history; null
>>>> UnavailableException()
>>>> at
>>>> org.apache.cassandra.thrift.Cassandra$truncate_result.read(Cassandra.j
>>>> ava:20212)
>>>> at
>>>> org.apache.cassandra.thrift.Cassandra$Client.recv_truncate(Cassandra.j
>>>> ava:1077)
>>>> at
>>>> org.apache.cassandra.thrift.Cassandra$Client.truncate(Cassandra.java:1
>>>> 052)
>>>> at
>>>> org.apache.cassandra.cli.CliClient.executeTruncate(CliClient.java:1445
>>>> )
>>>> at
>>>> org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:
>>>> 272)
>>>> at
>>>> org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.j
>>>> ava:220)
>>>> at org.apache.cassandra.cli.CliMain.main(CliMain.java:348)
>>>>
>>>>
>>>> So this looks that truncate goes very slow and too long, than
>>>> rpc_timeout_in_ms: 10000 (this can happens because we have very slow
>>>> disck on test machine)
>>>>
>>>> But in in cassandra system log i see follow exception:
>>>>
>>>>
>>>> ERROR [MutationStage:7022] 2012-05-17 12:19:14,356
>>>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>>>> Thread[MutationStage:7022,5,main]
>>>> java.io.IOError: java.io.IOException: unable to mkdirs
>>>>
>>> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-
>>> pur
>>>> chase_history
>>>> at
>>>>
>>> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(Column
>>> F
>>>> amilyStore.java:1433)
>>>> at
>>>>
>>> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.j
>>>> ava:1462)
>>>> at
>>>>
>>> org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.j
>>>> ava:1657)
>>>> at
>>>>
>>> org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandle
>>> r
>>>> .java:50)
>>>> at
>>>>
>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.j
>>>> ava:59)
>>>> at
>>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu
>>>> tor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>>> java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> Caused by: java.io.IOException: unable to mkdirs
>>>>
>>> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-
>>> pur
>>>> chase_history
>>>> at
>>>> org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:
>>>> 140)
>>>> at
>>>> org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:
>>>> 131)
>>>> at
>>>>
>>> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(Column
>>> F
>>>> amilyStore.java:1409)
>>>> ... 7 more
>>>>
>>>>
>>>> Also i see that in snapshort dir already exists
>>>> 1337242754356-purchase_history directory, so i think that snapshort
>>>> names that generate cassandra not uniquely.
>>>>
>>>> PS: We use cassandra 1.0.10 on Ubuntu 10.0.4-LTS
Re: Exception when truncate
Posted by ruslan usifov <ru...@gmail.com>.
Maybe, something changes in cassandra 1.0.x for truncate mechanism,
because in cassandra 0.8 truncate executes much faster on the same
data
2012/5/17 Viktor Jevdokimov <Vi...@adform.com>:
> Truncate flushes all memtables to free up commit logs, and that on all nodes. So this takes time. Discussed on this list not so long ago.
>
> Watch for:
> https://issues.apache.org/jira/browse/CASSANDRA-3651
> https://issues.apache.org/jira/browse/CASSANDRA-4006
>
>
>
> Best regards / Pagarbiai
>
> Viktor Jevdokimov
> Senior Developer
>
> Email: Viktor.Jevdokimov@adform.com
> Phone: +370 5 212 3063
> Fax: +370 5 261 0453
>
> J. Jasinskio 16C,
> LT-01112 Vilnius,
> Lithuania
>
>
>
> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.> -----Original Message-----
>> From: ruslan usifov [mailto:ruslan.usifov@gmail.com]
>> Sent: Thursday, May 17, 2012 13:06
>> To: user@cassandra.apache.org
>> Subject: Re: Exception when truncate
>>
>> Also i miss understand why on empty CF(no any SStable) truncate heavy
>> loads disk??
>>
>> 2012/5/17 ruslan usifov <ru...@gmail.com>:
>> > Hello
>> >
>> > I have follow situation on our test server:
>> >
>> > from cassandra-cli i try to use
>> >
>> > truncate purchase_history;
>> >
>> > 3 times i got:
>> >
>> > [default@township_6waves] truncate purchase_history; null
>> > UnavailableException()
>> > at
>> > org.apache.cassandra.thrift.Cassandra$truncate_result.read(Cassandra.j
>> > ava:20212)
>> > at
>> > org.apache.cassandra.thrift.Cassandra$Client.recv_truncate(Cassandra.j
>> > ava:1077)
>> > at
>> > org.apache.cassandra.thrift.Cassandra$Client.truncate(Cassandra.java:1
>> > 052)
>> > at
>> > org.apache.cassandra.cli.CliClient.executeTruncate(CliClient.java:1445
>> > )
>> > at
>> > org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:
>> > 272)
>> > at
>> > org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.j
>> > ava:220)
>> > at org.apache.cassandra.cli.CliMain.main(CliMain.java:348)
>> >
>> >
>> > So this looks that truncate goes very slow and too long, than
>> > rpc_timeout_in_ms: 10000 (this can happens because we have very slow
>> > disck on test machine)
>> >
>> > But in in cassandra system log i see follow exception:
>> >
>> >
>> > ERROR [MutationStage:7022] 2012-05-17 12:19:14,356
>> > AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> > Thread[MutationStage:7022,5,main]
>> > java.io.IOError: java.io.IOException: unable to mkdirs
>> >
>> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-
>> pur
>> > chase_history
>> > at
>> >
>> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(Column
>> F
>> > amilyStore.java:1433)
>> > at
>> >
>> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.j
>> > ava:1462)
>> > at
>> >
>> org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.j
>> > ava:1657)
>> > at
>> >
>> org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandle
>> r
>> > .java:50)
>> > at
>> >
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.j
>> > ava:59)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu
>> > tor.java:886)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>> > java:908)
>> > at java.lang.Thread.run(Thread.java:662)
>> > Caused by: java.io.IOException: unable to mkdirs
>> >
>> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-
>> pur
>> > chase_history
>> > at
>> > org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:
>> > 140)
>> > at
>> > org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:
>> > 131)
>> > at
>> >
>> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(Column
>> F
>> > amilyStore.java:1409)
>> > ... 7 more
>> >
>> >
>> > Also i see that in snapshort dir already exists
>> > 1337242754356-purchase_history directory, so i think that snapshort
>> > names that generate cassandra not uniquely.
>> >
>> > PS: We use cassandra 1.0.10 on Ubuntu 10.0.4-LTS
RE: Exception when truncate
Posted by Viktor Jevdokimov <Vi...@adform.com>.
Truncate flushes all memtables to free up commit logs, and that on all nodes. So this takes time. Discussed on this list not so long ago.
Watch for:
https://issues.apache.org/jira/browse/CASSANDRA-3651
https://issues.apache.org/jira/browse/CASSANDRA-4006
Best regards / Pagarbiai
Viktor Jevdokimov
Senior Developer
Email: Viktor.Jevdokimov@adform.com
Phone: +370 5 212 3063
Fax: +370 5 261 0453
J. Jasinskio 16C,
LT-01112 Vilnius,
Lithuania
Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.> -----Original Message-----
> From: ruslan usifov [mailto:ruslan.usifov@gmail.com]
> Sent: Thursday, May 17, 2012 13:06
> To: user@cassandra.apache.org
> Subject: Re: Exception when truncate
>
> Also i miss understand why on empty CF(no any SStable) truncate heavy
> loads disk??
>
> 2012/5/17 ruslan usifov <ru...@gmail.com>:
> > Hello
> >
> > I have follow situation on our test server:
> >
> > from cassandra-cli i try to use
> >
> > truncate purchase_history;
> >
> > 3 times i got:
> >
> > [default@township_6waves] truncate purchase_history; null
> > UnavailableException()
> > at
> > org.apache.cassandra.thrift.Cassandra$truncate_result.read(Cassandra.j
> > ava:20212)
> > at
> > org.apache.cassandra.thrift.Cassandra$Client.recv_truncate(Cassandra.j
> > ava:1077)
> > at
> > org.apache.cassandra.thrift.Cassandra$Client.truncate(Cassandra.java:1
> > 052)
> > at
> > org.apache.cassandra.cli.CliClient.executeTruncate(CliClient.java:1445
> > )
> > at
> > org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:
> > 272)
> > at
> > org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.j
> > ava:220)
> > at org.apache.cassandra.cli.CliMain.main(CliMain.java:348)
> >
> >
> > So this looks that truncate goes very slow and too long, than
> > rpc_timeout_in_ms: 10000 (this can happens because we have very slow
> > disck on test machine)
> >
> > But in in cassandra system log i see follow exception:
> >
> >
> > ERROR [MutationStage:7022] 2012-05-17 12:19:14,356
> > AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> > Thread[MutationStage:7022,5,main]
> > java.io.IOError: java.io.IOException: unable to mkdirs
> >
> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-
> pur
> > chase_history
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(Column
> F
> > amilyStore.java:1433)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.j
> > ava:1462)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.j
> > ava:1657)
> > at
> >
> org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandle
> r
> > .java:50)
> > at
> >
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.j
> > ava:59)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu
> > tor.java:886)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> > java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > Caused by: java.io.IOException: unable to mkdirs
> >
> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-
> pur
> > chase_history
> > at
> > org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:
> > 140)
> > at
> > org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:
> > 131)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(Column
> F
> > amilyStore.java:1409)
> > ... 7 more
> >
> >
> > Also i see that in snapshort dir already exists
> > 1337242754356-purchase_history directory, so i think that snapshort
> > names that generate cassandra not uniquely.
> >
> > PS: We use cassandra 1.0.10 on Ubuntu 10.0.4-LTS
Re: Exception when truncate
Posted by ruslan usifov <ru...@gmail.com>.
Also i miss understand why on empty CF(no any SStable) truncate heavy
loads disk??
2012/5/17 ruslan usifov <ru...@gmail.com>:
> Hello
>
> I have follow situation on our test server:
>
> from cassandra-cli i try to use
>
> truncate purchase_history;
>
> 3 times i got:
>
> [default@township_6waves] truncate purchase_history;
> null
> UnavailableException()
> at org.apache.cassandra.thrift.Cassandra$truncate_result.read(Cassandra.java:20212)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_truncate(Cassandra.java:1077)
> at org.apache.cassandra.thrift.Cassandra$Client.truncate(Cassandra.java:1052)
> at org.apache.cassandra.cli.CliClient.executeTruncate(CliClient.java:1445)
> at org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:272)
> at org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:220)
> at org.apache.cassandra.cli.CliMain.main(CliMain.java:348)
>
>
> So this looks that truncate goes very slow and too long, than
> rpc_timeout_in_ms: 10000 (this can happens because we have very slow
> disck on test machine)
>
> But in in cassandra system log i see follow exception:
>
>
> ERROR [MutationStage:7022] 2012-05-17 12:19:14,356
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[MutationStage:7022,5,main]
> java.io.IOError: java.io.IOException: unable to mkdirs
> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-purchase_history
> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1433)
> at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1462)
> at org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.java:1657)
> at org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandler.java:50)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: unable to mkdirs
> /home/cassandra/1.0.0/data/township_6waves/snapshots/1337242754356-purchase_history
> at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:140)
> at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:131)
> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1409)
> ... 7 more
>
>
> Also i see that in snapshort dir already exists
> 1337242754356-purchase_history directory, so i think that snapshort
> names that generate cassandra not uniquely.
>
> PS: We use cassandra 1.0.10 on Ubuntu 10.0.4-LTS