You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Elias Ross <ge...@noderunner.net> on 2013/11/04 19:08:28 UTC
Duplicate hard link - Cassandra 1.2.9
Cassandra 1.2.9, embedded into the RHQ 4.9 project.
I'm getting the following:
Caused by: java.lang.RuntimeException: Tried to create duplicate hard link
to
/data05/rhq/data/system/NodeIdInfo/snapshots/1383587405678/system-NodeIdInfo-ic-
1-TOC.txt
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:70)
at
org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:1081)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1567)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1612)
at org.apache.cassandra.db.Table.snapshot(Table.java:194)
at
org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:2203)
Clearing the snapshot directory doesn't seem to fix this issue. Restarting
the node doesn't fix it. This is obviously a bug, but I'm not sure what to
do about it.
I don't have enough context to know what this means or what to do to fix it.
Re: Duplicate hard link - Cassandra 1.2.9
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Nov 11, 2013 at 8:06 PM, Aaron Morton <aa...@thelastpickle.com>wrote:
> If you can reproduce it may be time to raise a ticket including the JBOD
> setup https://issues.apache.org/jira/browse/CASSANDRA
>
Per OP on https://issues.apache.org/jira/browse/CASSANDRA-6298
"
As it turns out, I was using symlinks. I had my last data directory
pointing to the same data directory.
lrwxrwxrwx 1 root root 6 Oct 30 18:37 data01 -> /data1
lrwxrwxrwx 1 root root 6 Oct 30 18:37 data02 -> /data2
lrwxrwxrwx 1 root root 6 Oct 30 18:37 data03 -> /data3
lrwxrwxrwx 1 root root 6 Oct 30 18:37 data04 -> /data4
lrwxrwxrwx 1 root root 6 Oct 30 18:37 data05 -> /data5
lrwxrwxrwx 1 root root 6 Nov 8 18:46 data06 -> /data5
As you can see (not so clearly) the last link data06 is pointing to data5.
Maybe Cassandra can do some basic checking (at startup) to check that the
data directories aren't repeated or pointing to the same location.
In any case, not a bug in the server.
"
=Rob
Re: Duplicate hard link - Cassandra 1.2.9
Posted by Aaron Morton <aa...@thelastpickle.com>.
> With the scrub --no-snapshots option I get:
>
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.FileNotFoundException: /data06/rhq/data/system/schema_keyspaces/system-schema_keyspaces-ic-20-Data.db
> (No such file or directory)
Can you paste the full error from the system.log (it includes the thread).
Sounds like it could be an issue with scrub / snapshot and JBOD.
If you can reproduce it may be time to raise a ticket including the JBOD setup https://issues.apache.org/jira/browse/CASSANDRA
Cheers
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 8/11/2013, at 2:21 pm, Elias Ross <ge...@noderunner.net> wrote:
>
>
>
> On Thu, Nov 7, 2013 at 5:08 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Thu, Nov 7, 2013 at 8:23 AM, Elias Ross <ge...@gmail.com> wrote:
>
> I've been attempting a scrub and see the same issue. Is there some mapping issue? Are six data directories too much for Cassandra?
>
> Do both filesystems support hard links?
>
>
> I guess so. This is my mount configuration. (note: /data01 links to /data1 etc.)
>
> /dev/sdb1 on /data1 type ext3 (rw)
> /dev/sdc1 on /data2 type ext3 (rw)
> /dev/sdd1 on /data3 type ext3 (rw)
> /dev/sde1 on /data4 type ext3 (rw)
> /dev/sdf1 on /data5 type ext3 (rw)
> /dev/sdg1 on /data6 type ext3 (rw)
>
>
> How do you have JBOD defined?
>
> cassandra.yaml:
>
> partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> data_file_directories:
> - /data01/rhq/data
> - /data02/rhq/data
> - /data03/rhq/data
> - /data04/rhq/data
> - /data05/rhq/data
> - /data06/rhq/data
>
Re: Duplicate hard link - Cassandra 1.2.9
Posted by Elias Ross <ge...@noderunner.net>.
On Thu, Nov 7, 2013 at 5:08 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Thu, Nov 7, 2013 at 8:23 AM, Elias Ross <ge...@gmail.com> wrote:
>
>>
>> I've been attempting a scrub and see the same issue. Is there some
>> mapping issue? Are six data directories too much for Cassandra?
>>
>
> Do both filesystems support hard links?
>
>
I guess so. This is my mount configuration. (note: /data01 links to /data1
etc.)
/dev/sdb1 on /data1 type ext3 (rw)
/dev/sdc1 on /data2 type ext3 (rw)
/dev/sdd1 on /data3 type ext3 (rw)
/dev/sde1 on /data4 type ext3 (rw)
/dev/sdf1 on /data5 type ext3 (rw)
/dev/sdg1 on /data6 type ext3 (rw)
> How do you have JBOD defined?
>
cassandra.yaml:
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /data01/rhq/data
- /data02/rhq/data
- /data03/rhq/data
- /data04/rhq/data
- /data05/rhq/data
- /data06/rhq/data
Re: Duplicate hard link - Cassandra 1.2.9
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Nov 7, 2013 at 8:23 AM, Elias Ross <ge...@gmail.com> wrote:
>
> I've been attempting a scrub and see the same issue. Is there some mapping
> issue? Are six data directories too much for Cassandra?
>
Do both filesystems support hard links?
How do you have JBOD defined?
=Rob
Re: Duplicate hard link - Cassandra 1.2.9
Posted by Elias Ross <ge...@gmail.com>.
Last time to respond to myself...
On Thu, Nov 7, 2013 at 8:23 AM, Elias Ross <ge...@gmail.com> wrote:
I've been attempting a scrub and see the same issue. Is there some mapping
> issue? Are six data directories too much for Cassandra?
>
>
With the scrub --no-snapshots option I get:
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.io.FileNotFoundException:
/data06/rhq/data/system/schema_keyspaces/system-schema_keyspaces-ic-20-Data.db
(No such file or directory)
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:274)
at
org.apache.cassandra.db.compaction.CompactionManager.performScrub(CompactionManager.java:279)
at
org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:975)
at
org.apache.cassandra.service.StorageService.scrub(StorageService.java:2155)
...
Caused by: java.io.FileNotFoundException:
/data06/rhq/data/system/schema_keyspaces/system-schema_keyspaces-ic-20-Data.db
(No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
at
org.apache.cassandra.io.util.ThrottledReader.<init>(ThrottledReader.java:35)
at
org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
... 13 more
The schema files are on a different drive:
[rhq@st11p01ad-rhq006 bin]$ locate system-schema
...
/data05/rhq/data/system/schema_keyspaces/system-schema_keyspaces-ic-20-Data.db
/data05/rhq/data/system/schema_keyspaces/system-schema_keyspaces-ic-20-Digest.sha1
/data05/rhq/data/system/schema_keyspaces/system-schema_keyspaces-ic-20-Filter.db
/data05/rhq/data/system/schema_keyspaces/system-schema_keyspaces-ic-20-Index.db
/data05/rhq/data/system/schema_keyspaces/system-schema_keyspaces-ic-20-Statistics.db
/data05/rhq/data/system/schema_keyspaces/system-schema_keyspaces-ic-20-Summary.db
/data05/rhq/data/system/schema_keyspaces/system-schema_keyspaces-ic-20-TOC.txt
Re: Duplicate hard link - Cassandra 1.2.9
Posted by Elias Ross <ge...@gmail.com>.
On Mon, Nov 4, 2013 at 2:09 PM, Elias Ross <ge...@gmail.com> wrote:
>
> Thanks Robert.
>
> CASSANDRA-6298
>
>
I've been attempting a scrub and see the same issue. Is there some mapping
issue? Are six data directories too much for Cassandra?
$ ./nodetool -p 7299 scrub rhq
Exception in thread "main" java.lang.RuntimeException: Tried to create
duplicate hard link to
/data05/rhq/data/rhq/six_hour_metrics/snapshots/pre-scrub-1383787540489/rhq-six_hour
_metrics-ic-6-Summary.db
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:70)
at
org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:1081)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1567)
at
org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:974)
at
org.apache.cassandra.service.StorageService.scrub(StorageService.java:2155)
Re: Duplicate hard link - Cassandra 1.2.9
Posted by Elias Ross <ge...@gmail.com>.
Thanks Robert.
CASSANDRA-6298
Is there any way to maybe do a workaround? I guess the thinking I have is
the duplicate hard link is probably pretty harmless and getting rid of the
check would at least get me past this issue.
Re: Duplicate hard link - Cassandra 1.2.9
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Nov 4, 2013 at 10:08 AM, Elias Ross <ge...@noderunner.net> wrote:
> Cassandra 1.2.9, embedded into the RHQ 4.9 project.
>
> I'm getting the following:
>
> Caused by: java.lang.RuntimeException: Tried to create duplicate hard link
> to
> /data05/rhq/data/system/NodeIdInfo/snapshots/1383587405678/system-NodeIdInfo-ic-
>
Someone else having a similar issue, but on upgrade to 2.0.x from 1.2.10.
http://mail-archives.apache.org/mod_mbox/cassandra-user/201309.mbox/%3C00b801ceb9ef$e3cc6a70$ab653f50$@struq.com%3E
In that case, it was : https://issues.apache.org/jira/browse/CASSANDRA-6093
But I'm not sure that applies to your issue? If not, file a JIRA! :D
=Rob