You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by André Cruz <an...@co.sapo.pt> on 2012/05/07 23:00:18 UTC
Error deleting column families with 1.1
Hello.
Since I upgraded to Cassandra 1.1, I get the following error when
trying to delete a CF. After this happens the CF is not accessible
anymore, but I cannot create another one with the same name until I
restart the server.
INFO [MigrationStage:1] 2012-05-07 18:10:12,682 ColumnFamilyStore.java
(line 634) Enqueuing flush of
Memtable-schema_columnfamilies@1128094887(978/1222 serialized/live
bytes, 21 ops)
INFO [FlushWriter:2] 2012-05-07 18:10:12,682 Memtable.java (line 266)
Writing Memtable-schema_columnfamilies@1128094887(978/1222
serialized/live bytes, 21 ops)
INFO [FlushWriter:2] 2012-05-07 18:10:12,720 Memtable.java (line 307)
Completed flushing
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db (1041
bytes)
INFO [MigrationStage:1] 2012-05-07 18:10:12,721 ColumnFamilyStore.java
(line 634) Enqueuing flush of
Memtable-schema_columns@1599271050(392/490 serialized/live bytes, 8 ops)
INFO [FlushWriter:2] 2012-05-07 18:10:12,722 Memtable.java (line 266)
Writing Memtable-schema_columns@1599271050(392/490 serialized/live
bytes, 8 ops)
INFO [CompactionExecutor:8] 2012-05-07 18:10:12,722
CompactionTask.java (line 114) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-26-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfam
ilies-hc-27-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-25-Data.db')]
INFO [FlushWriter:2] 2012-05-07 18:10:12,806 Memtable.java (line 307)
Completed flushing
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-23-Data.db (447
bytes)
INFO [CompactionExecutor:8] 2012-05-07 18:10:12,811
CompactionTask.java (line 225) Compacted to
[/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-29-Data.db,]. 24,797 to
21,431
(~86% of original) bytes for 2 keys at 0.232252MB/s. Time: 88ms.
ERROR [MigrationStage:1] 2012-05-07 18:10:12,895 CLibrary.java (line
158) Unable to create hard link
com.sun.jna.LastErrorException: errno was 17
at org.apache.cassandra.utils.CLibrary.link(Native Method)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:150)
at
org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at
org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [Thrift:17] 2012-05-07 18:10:12,898 CustomTThreadPoolServer.java
(line 204) Error occurred during processing of message.
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.io.IOError: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/
Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:372)
at
org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:191)
at
org.apache.cassandra.service.MigrationManager.announceColumnFamilyDrop(MigrationManager.java:182)
at
org.apache.cassandra.thrift.CassandraServer.system_drop_column_family(CassandraServer.java:948)
at
org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3348)
at
org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3336)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException: java.io.IOError:
java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
... 11 more
Caused by: java.io.IOError: java.io.IOException: Unable to create hard
link from /var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at
org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
at
org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
... 10 more
ERROR [MigrationStage:1] 2012-05-07 18:10:12,899
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[MigrationStage:1,5,main]
java.io.IOError: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at
org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
at
org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
... 10 more
I've tried recreating the data dirs, to see if this was some kind of
permissions problem, but the error happens every time and with any CF.
Has anyone seen this error?
I'm using the Cassandra Debian package on Debian squeeze and the Sun
JVM (build 1.6.0_26-b03).
Best regards,
André
Re: Error deleting column families with 1.1
Posted by Data Craftsman <da...@gmail.com>.
We have similar issue. I'll try to reproduce it and upload logs soon.
On Wed, May 9, 2012 at 3:30 AM, André Cruz <an...@co.sapo.pt> wrote:
> Here it is: https://issues.apache.org/jira/browse/CASSANDRA-4230
>
> Please let me know if you need further info.
>
> Best regards,
> André
>
> On May 8, 2012, at 23:55 , aaron morton wrote:
>
> Could you please create a ticket for this
> on https://issues.apache.org/jira/browse/CASSANDRA
>
> Please include:
> * operating system
> * keyspace / column family definition
> * output of of "ls -lah" for the "/var/lib/cassandra/data/Disco/Client/"
> directory after the error occurs.
>
> Thanks
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 8/05/2012, at 9:00 AM, André Cruz wrote:
>
> Hello.
>
> Since I upgraded to Cassandra 1.1, I get the following error when trying to
> delete a CF. After this happens the CF is not accessible anymore, but I
> cannot create another one with the same name until I restart the server.
>
> INFO [MigrationStage:1] 2012-05-07 18:10:12,682 ColumnFamilyStore.java (line
> 634) Enqueuing flush of Memtable-schema_columnfamilies@1128094887(978/1222
> serialized/live bytes, 21 ops)
> INFO [FlushWriter:2] 2012-05-07 18:10:12,682 Memtable.java (line 266)
> Writing Memtable-schema_columnfamilies@1128094887(978/1222 serialized/live
> bytes, 21 ops)
> INFO [FlushWriter:2] 2012-05-07 18:10:12,720 Memtable.java (line 307)
> Completed flushing
> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db
> (1041 bytes)
> INFO [MigrationStage:1] 2012-05-07 18:10:12,721 ColumnFamilyStore.java (line
> 634) Enqueuing flush of Memtable-schema_columns@1599271050(392/490
> serialized/live bytes, 8 ops)
> INFO [FlushWriter:2] 2012-05-07 18:10:12,722 Memtable.java (line 266)
> Writing Memtable-schema_columns@1599271050(392/490 serialized/live bytes, 8
> ops)
> INFO [CompactionExecutor:8] 2012-05-07 18:10:12,722 CompactionTask.java
> (line 114) Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-26-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfam
> ilies-hc-27-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-25-Data.db')]
> INFO [FlushWriter:2] 2012-05-07 18:10:12,806 Memtable.java (line 307)
> Completed flushing
> /var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-23-Data.db
> (447 bytes)
> INFO [CompactionExecutor:8] 2012-05-07 18:10:12,811 CompactionTask.java
> (line 225) Compacted to
> [/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-29-Data.db,].
> 24,797 to 21,431
> (~86% of original) bytes for 2 keys at 0.232252MB/s. Time: 88ms.
> ERROR [MigrationStage:1] 2012-05-07 18:10:12,895 CLibrary.java (line 158)
> Unable to create hard link
> com.sun.jna.LastErrorException: errno was 17
> at org.apache.cassandra.utils.CLibrary.link(Native Method)
> at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:150)
> at
> org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
> at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
> at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
> at
> org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thrift:17] 2012-05-07 18:10:12,898 CustomTThreadPoolServer.java (line
> 204) Error occurred during processing of message.
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.io.IOError: java.io.IOException: Unable to create hard link from
> /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/
> Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:372)
> at
> org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:191)
> at
> org.apache.cassandra.service.MigrationManager.announceColumnFamilyDrop(MigrationManager.java:182)
> at
> org.apache.cassandra.thrift.CassandraServer.system_drop_column_family(CassandraServer.java:948)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3348)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3336)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException: java.io.IOError:
> java.io.IOException: Unable to create hard link from
> /var/lib/cassandra/data/Disco/Client/Client.json to
> /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json
> (errno 17)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
> ... 11 more
> Caused by: java.io.IOError: java.io.IOException: Unable to create hard link
> from /var/lib/cassandra/data/Disco/Client/Client.json to
> /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json
> (errno 17)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
> at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
> at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
> at
> org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> Caused by: java.io.IOException: Unable to create hard link from
> /var/lib/cassandra/data/Disco/Client/Client.json to
> /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json
> (errno 17)
> at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
> at
> org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
> ... 10 more
> ERROR [MigrationStage:1] 2012-05-07 18:10:12,899
> AbstractCassandraDaemon.java (line 134) Exception in thread
> Thread[MigrationStage:1,5,main]
> java.io.IOError: java.io.IOException: Unable to create hard link from
> /var/lib/cassandra/data/Disco/Client/Client.json to
> /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json
> (errno 17)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
> at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
> at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
> at
> org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Unable to create hard link from
> /var/lib/cassandra/data/Disco/Client/Client.json to
> /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json
> (errno 17)
> at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
> at
> org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
> ... 10 more
>
>
>
> I've tried recreating the data dirs, to see if this was some kind of
> permissions problem, but the error happens every time and with any CF. Has
> anyone seen this error?
>
> I'm using the Cassandra Debian package on Debian squeeze and the Sun JVM
> (build 1.6.0_26-b03).
>
> Best regards,
> André
>
>
>
--
--
Thanks,
Charlie (@mujiang) 一个 木匠
=======
Data Architect Developer
http://mujiang.blogspot.com
Re: Error deleting column families with 1.1
Posted by André Cruz <an...@co.sapo.pt>.
Here it is: https://issues.apache.org/jira/browse/CASSANDRA-4230
Please let me know if you need further info.
Best regards,
André
On May 8, 2012, at 23:55 , aaron morton wrote:
> Could you please create a ticket for this on https://issues.apache.org/jira/browse/CASSANDRA
>
> Please include:
> * operating system
> * keyspace / column family definition
> * output of of "ls -lah" for the "/var/lib/cassandra/data/Disco/Client/" directory after the error occurs.
>
> Thanks
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 8/05/2012, at 9:00 AM, André Cruz wrote:
>
>> Hello.
>>
>> Since I upgraded to Cassandra 1.1, I get the following error when trying to delete a CF. After this happens the CF is not accessible anymore, but I cannot create another one with the same name until I restart the server.
>>
>> INFO [MigrationStage:1] 2012-05-07 18:10:12,682 ColumnFamilyStore.java (line 634) Enqueuing flush of Memtable-schema_columnfamilies@1128094887(978/1222 serialized/live bytes, 21 ops)
>> INFO [FlushWriter:2] 2012-05-07 18:10:12,682 Memtable.java (line 266) Writing Memtable-schema_columnfamilies@1128094887(978/1222 serialized/live bytes, 21 ops)
>> INFO [FlushWriter:2] 2012-05-07 18:10:12,720 Memtable.java (line 307) Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db (1041 bytes)
>> INFO [MigrationStage:1] 2012-05-07 18:10:12,721 ColumnFamilyStore.java (line 634) Enqueuing flush of Memtable-schema_columns@1599271050(392/490 serialized/live bytes, 8 ops)
>> INFO [FlushWriter:2] 2012-05-07 18:10:12,722 Memtable.java (line 266) Writing Memtable-schema_columns@1599271050(392/490 serialized/live bytes, 8 ops)
>> INFO [CompactionExecutor:8] 2012-05-07 18:10:12,722 CompactionTask.java (line 114) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-26-Data.db'),
>> SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfam
>> ilies-hc-27-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-25-Data.db')]
>> INFO [FlushWriter:2] 2012-05-07 18:10:12,806 Memtable.java (line 307) Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-23-Data.db (447 bytes)
>> INFO [CompactionExecutor:8] 2012-05-07 18:10:12,811 CompactionTask.java (line 225) Compacted to [/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-29-Data.db,]. 24,797 to 21,431
>> (~86% of original) bytes for 2 keys at 0.232252MB/s. Time: 88ms.
>> ERROR [MigrationStage:1] 2012-05-07 18:10:12,895 CLibrary.java (line 158) Unable to create hard link
>> com.sun.jna.LastErrorException: errno was 17
>> at org.apache.cassandra.utils.CLibrary.link(Native Method)
>> at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:150)
>> at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
>> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
>> at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
>> at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
>> at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
>> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
>> at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> ERROR [Thrift:17] 2012-05-07 18:10:12,898 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/
>> Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
>> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:372)
>> at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:191)
>> at org.apache.cassandra.service.MigrationManager.announceColumnFamilyDrop(MigrationManager.java:182)
>> at org.apache.cassandra.thrift.CassandraServer.system_drop_column_family(CassandraServer.java:948)
>> at org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3348)
>> at org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3336)
>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
>> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
>> ... 11 more
>> Caused by: java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
>> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
>> at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
>> at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
>> at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
>> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
>> at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> ... 3 more
>> Caused by: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
>> at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
>> at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
>> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
>> ... 10 more
>> ERROR [MigrationStage:1] 2012-05-07 18:10:12,899 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]
>> java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
>> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
>> at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
>> at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
>> at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
>> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
>> at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
>> at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
>> at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
>> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
>> ... 10 more
>>
>>
>>
>> I've tried recreating the data dirs, to see if this was some kind of permissions problem, but the error happens every time and with any CF. Has anyone seen this error?
>>
>> I'm using the Cassandra Debian package on Debian squeeze and the Sun JVM (build 1.6.0_26-b03).
>>
>> Best regards,
>> André
>>
>
Re: Error deleting column families with 1.1
Posted by aaron morton <aa...@thelastpickle.com>.
Could you please create a ticket for this on https://issues.apache.org/jira/browse/CASSANDRA
Please include:
* operating system
* keyspace / column family definition
* output of of "ls -lah" for the "/var/lib/cassandra/data/Disco/Client/" directory after the error occurs.
Thanks
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 8/05/2012, at 9:00 AM, André Cruz wrote:
> Hello.
>
> Since I upgraded to Cassandra 1.1, I get the following error when trying to delete a CF. After this happens the CF is not accessible anymore, but I cannot create another one with the same name until I restart the server.
>
> INFO [MigrationStage:1] 2012-05-07 18:10:12,682 ColumnFamilyStore.java (line 634) Enqueuing flush of Memtable-schema_columnfamilies@1128094887(978/1222 serialized/live bytes, 21 ops)
> INFO [FlushWriter:2] 2012-05-07 18:10:12,682 Memtable.java (line 266) Writing Memtable-schema_columnfamilies@1128094887(978/1222 serialized/live bytes, 21 ops)
> INFO [FlushWriter:2] 2012-05-07 18:10:12,720 Memtable.java (line 307) Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db (1041 bytes)
> INFO [MigrationStage:1] 2012-05-07 18:10:12,721 ColumnFamilyStore.java (line 634) Enqueuing flush of Memtable-schema_columns@1599271050(392/490 serialized/live bytes, 8 ops)
> INFO [FlushWriter:2] 2012-05-07 18:10:12,722 Memtable.java (line 266) Writing Memtable-schema_columns@1599271050(392/490 serialized/live bytes, 8 ops)
> INFO [CompactionExecutor:8] 2012-05-07 18:10:12,722 CompactionTask.java (line 114) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-26-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfam
> ilies-hc-27-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-25-Data.db')]
> INFO [FlushWriter:2] 2012-05-07 18:10:12,806 Memtable.java (line 307) Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-23-Data.db (447 bytes)
> INFO [CompactionExecutor:8] 2012-05-07 18:10:12,811 CompactionTask.java (line 225) Compacted to [/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-29-Data.db,]. 24,797 to 21,431
> (~86% of original) bytes for 2 keys at 0.232252MB/s. Time: 88ms.
> ERROR [MigrationStage:1] 2012-05-07 18:10:12,895 CLibrary.java (line 158) Unable to create hard link
> com.sun.jna.LastErrorException: errno was 17
> at org.apache.cassandra.utils.CLibrary.link(Native Method)
> at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:150)
> at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
> at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
> at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
> at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
> at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thrift:17] 2012-05-07 18:10:12,898 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/
> Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:372)
> at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:191)
> at org.apache.cassandra.service.MigrationManager.announceColumnFamilyDrop(MigrationManager.java:182)
> at org.apache.cassandra.thrift.CassandraServer.system_drop_column_family(CassandraServer.java:948)
> at org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3348)
> at org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3336)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
> ... 11 more
> Caused by: java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
> at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
> at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
> at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
> at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> Caused by: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
> at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
> at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
> ... 10 more
> ERROR [MigrationStage:1] 2012-05-07 18:10:12,899 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]
> java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
> at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
> at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
> at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
> at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
> at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
> at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
> at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
> ... 10 more
>
>
>
> I've tried recreating the data dirs, to see if this was some kind of permissions problem, but the error happens every time and with any CF. Has anyone seen this error?
>
> I'm using the Cassandra Debian package on Debian squeeze and the Sun JVM (build 1.6.0_26-b03).
>
> Best regards,
> André
>