You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Christopher Wirt <ch...@struq.com> on 2013/09/25 01:48:08 UTC
1.2.10 -> 2.0.1 migration issue
Hi,
Just had a go at upgrading a node to the latest stable c* 2 release and
think I ran into some issues with manifest migration.
On initial start up I hit this error as it starts to load the first of my
CF.
INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2
INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration
ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
Exception encountered during startup
FSWriteError in
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
42)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
Caused by: java.nio.file.NoSuchFileException:
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json ->
/disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474
)
at java.nio.file.Files.createLink(Files.java:1037)
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)
... 5 more
I had already successful run a test migration on our dev server. Only real
difference I can see if the number of data directories defined and the
amount of data being held.
I've run upgradesstables under 1.2.10. I have always been using vnodes and
CQL3. I recently moved to using LZ4 instead of Snappy..
I tried to startup again and it gave me a slightly different error
INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2
INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration
ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
Exception encountered during startup
java.lang.RuntimeException: Tried to create duplicate hard link to
/disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:129)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
42)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
Will have a go recreating this tomorrow.
Any insight or guesses at what the issue might be are always welcome.
Thanks,
Chris
RE: 1.2.10 -> 2.0.1 migration issue
Posted by Christopher Wirt <ch...@struq.com>.
Hi Marcus,
I've seen your patch. This works with what I'm seeing. The first data
directory only contained the JSON manifest at that time.
As a workaround I've made sure that each of the snapshot directories now
exist before starting up.
I still end up with the second exception I posted regarding a duplicate hard
link. Possibly two unrelated exceptions.
After getting this error. Looking at the datadirs
Data1 contains
JSON manifests
Loads of data files
Snapshot directory
Data2 contains
Just the snapshot directory
Data3 contains
Just the snapshot directory
INFO 12:56:22,766 Migrating manifest for struqrealtime/impressionstorev2
INFO 12:56:22,767 Snapshotting struqrealtime, impressionstorev2 to
pre-sstablemetamigration
ERROR 12:56:22,787 Exception encountered during startup
java.lang.RuntimeException: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
43)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
java.lang.RuntimeException: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
43)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
Exception encountered during startup: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json
Thanks,
Chris
From: Marcus Eriksson [mailto:krummas@gmail.com]
Sent: 25 September 2013 13:11
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue
you are probably reading trunk NEWS.txt
read the ticket for explanation of what the issue was (it is a proper bug)
On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt <ch...@struq.com>
wrote:
Hi Marcus,
Thanks for having a look at this.
Just noticed this in the NEWS.txt
For leveled compaction users, 2.0 must be atleast started before
upgrading to 2.1 due to the fact that the old JSON leveled
manifest is migrated into the sstable metadata files on startup
in 2.0 and this code is gone from 2.1.
Basically, my fault for skimming over this too quickly.
We will move from 1.2.10 -> 2.0 -> 2.1
Thanks,
Chris
From: Marcus Eriksson [mailto:krummas@gmail.com]
Sent: 25 September 2013 09:37
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue
cant really reproduce, could you update the ticket with a bit more info
about your setup?
do you have multiple .json files in your data dirs?
On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson <kr...@gmail.com> wrote:
this is most likely a bug, filed
https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a
look today.
On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt <ch...@struq.com>
wrote:
Hi,
Just had a go at upgrading a node to the latest stable c* 2 release and
think I ran into some issues with manifest migration.
On initial start up I hit this error as it starts to load the first of my
CF.
INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2
INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration
ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
Exception encountered during startup
FSWriteError in
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
42)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
Caused by: java.nio.file.NoSuchFileException:
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json ->
/disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474
)
at java.nio.file.Files.createLink(Files.java:1037)
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)
... 5 more
I had already successful run a test migration on our dev server. Only real
difference I can see if the number of data directories defined and the
amount of data being held.
I've run upgradesstables under 1.2.10. I have always been using vnodes and
CQL3. I recently moved to using LZ4 instead of Snappy..
I tried to startup again and it gave me a slightly different error
INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2
INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration
ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
Exception encountered during startup
java.lang.RuntimeException: Tried to create duplicate hard link to
/disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:129)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
42)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
Will have a go recreating this tomorrow.
Any insight or guesses at what the issue might be are always welcome.
Thanks,
Chris
Re: 1.2.10 -> 2.0.1 migration issue
Posted by Marcus Eriksson <kr...@gmail.com>.
you probably have to remove the old snapshots before trying to restart
On Wed, Sep 25, 2013 at 3:05 PM, Christopher Wirt <ch...@struq.com>wrote:
> Should also say. I have managed to move one node from 1.2.10 to 2.0.0. I’m
> seeing this error on the machine I tried to migrate earlier to 2.0.1****
>
> ** **
>
> Thanks****
>
> ** **
>
> *From:* Christopher Wirt [mailto:chris.wirt@struq.com]
> *Sent:* 25 September 2013 14:04
> *To:* 'user@cassandra.apache.org'
> *Subject:* RE: 1.2.10 -> 2.0.1 migration issue****
>
> ** **
>
> Hi Marcus,****
>
> ** **
>
> I’ve seen your patch. This works with what I’m seeing. The first data
> directory only contained the JSON manifest at that time.****
>
> ** **
>
> As a workaround I’ve made sure that each of the snapshot directories now
> exist before starting up.****
>
> ** **
>
> I still end up with the second exception I posted regarding a duplicate
> hard link. Possibly two unrelated exceptions.****
>
> ** **
>
> After getting this error. Looking at the datadirs****
>
> Data1 contains ****
>
> JSON manifests****
>
> Loads of data files****
>
> Snapshot directory****
>
> Data2 contains****
>
> Just the snapshot directory****
>
> Data3 contains****
>
> Just the snapshot directory****
>
> ** **
>
> INFO 12:56:22,766 Migrating manifest for struqrealtime/impressionstorev2**
> **
>
> INFO 12:56:22,767 Snapshotting struqrealtime, impressionstorev2 to
> pre-sstablemetamigration****
>
> ERROR 12:56:22,787 Exception encountered during startup****
>
> java.lang.RuntimeException: Tried to create duplicate hard link to
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ****
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
> ****
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
> ****
>
> java.lang.RuntimeException: Tried to create duplicate hard link to
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ****
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
> ****
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
> ****
>
> Exception encountered during startup: Tried to create duplicate hard link
> to
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ****
>
> ** **
>
> Thanks,****
>
> ** **
>
> Chris****
>
> ** **
>
> ** **
>
> *From:* Marcus Eriksson [mailto:krummas@gmail.com]
> *Sent:* 25 September 2013 13:11
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: 1.2.10 -> 2.0.1 migration issue****
>
> ** **
>
> you are probably reading trunk NEWS.txt****
>
> ** **
>
> read the ticket for explanation of what the issue was (it is a proper bug)
> ****
>
> ** **
>
> On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt <ch...@struq.com>
> wrote:****
>
> Hi Marcus,****
>
> Thanks for having a look at this.****
>
> ****
>
> Just noticed this in the NEWS.txt ****
>
> ****
>
> For *leveled* compaction users, 2.0 must be *atleast* started before****
>
> upgrading to 2.1 due to the fact that the old JSON *leveled*****
>
> manifest is migrated into the *sstable* *metadata* files on startup**
> **
>
> in 2.0 and this code is gone from 2.1.****
>
> ****
>
> Basically, my fault for skimming over this too quickly. ****
>
> ****
>
> We will move from 1.2.10 -> 2.0 -> 2.1****
>
> ****
>
> Thanks,****
>
> Chris****
>
> ****
>
> ****
>
> *From:* Marcus Eriksson [mailto:krummas@gmail.com]
> *Sent:* 25 September 2013 09:37
> *To:* user@cassandra.apache.org
> *Subject:* Re: 1.2.10 -> 2.0.1 migration issue****
>
> ****
>
> cant really reproduce, could you update the ticket with a bit more info
> about your setup?****
>
> ****
>
> do you have multiple .json files in your data dirs?****
>
> ****
>
> On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson <kr...@gmail.com>
> wrote:****
>
> this is most likely a bug, filed
> https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have
> a look today.****
>
> ****
>
> On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt <ch...@struq.com>
> wrote:****
>
> Hi,****
>
> ****
>
> Just had a go at upgrading a node to the latest stable c* 2 release and
> think I ran into some issues with manifest migration.****
>
> ****
>
> On initial start up I hit this error as it starts to load the first of my
> CF. ****
>
> ****
>
> INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
> Migrating manifest for struqrealtime/impressionstorev2****
>
> INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration*
> ***
>
> ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
> Exception encountered during startup****
>
> FSWriteError in
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ****
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
> ****
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> ****
>
> Caused by: java.nio.file.NoSuchFileException:
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ->
> /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json
> ****
>
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)****
>
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)****
>
> at
> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474)
> ****
>
> at java.nio.file.Files.createLink(Files.java:1037)****
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)**
> **
>
> ... 5 more****
>
> ****
>
> I had already successful run a test migration on our dev server. Only real
> difference I can see if the number of data directories defined and the
> amount of data being held. ****
>
> ****
>
> I’ve run upgradesstables under 1.2.10. I have always been using vnodes and
> CQL3. I recently moved to using LZ4 instead of Snappy..****
>
> ****
>
> I tried to startup again and it gave me a slightly different error****
>
> ****
>
> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
> Migrating manifest for struqrealtime/impressionstorev2****
>
> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration*
> ***
>
> ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
> Exception encountered during startup****
>
> java.lang.RuntimeException: Tried to create duplicate hard link to
> /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt
> ****
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129)
> ****
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> ****
>
> ****
>
> Will have a go recreating this tomorrow.****
>
> ****
>
> Any insight or guesses at what the issue might be are always welcome.****
>
> ****
>
> Thanks,****
>
> Chris****
>
> ****
>
> ****
>
> ** **
>
RE: 1.2.10 -> 2.0.1 migration issue
Posted by Christopher Wirt <ch...@struq.com>.
Yes that was the problem.
I got confused between 2.0.1 and 2.1.0 after downloading the trunk source.
From: Robert Coli [mailto:rcoli@eventbrite.com]
Sent: 25 September 2013 18:10
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue
On Wed, Sep 25, 2013 at 6:05 AM, Christopher Wirt <ch...@struq.com>
wrote:
Should also say. I have managed to move one node from 1.2.10 to 2.0.0. I'm
seeing this error on the machine I tried to migrate earlier to 2.0.1
I'm confused... for the record :
1) you tried to upgrade from 1.2.10 to 2.0.1
2) the NEWS.txt snippet you posted refers to upgrading from versions below
2.0 to 2.1
3) 2.0.1 is 2.0, not 2.1
Therefore the problem is actually
https://issues.apache.org/jira/browse/CASSANDRA-6093 ?
=Rob
Re: 1.2.10 -> 2.0.1 migration issue
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Sep 25, 2013 at 6:05 AM, Christopher Wirt <ch...@struq.com>wrote:
> Should also say. I have managed to move one node from 1.2.10 to 2.0.0. I’m
> seeing this error on the machine I tried to migrate earlier to 2.0.1
>
I'm confused... for the record :
1) you tried to upgrade from 1.2.10 to 2.0.1
2) the NEWS.txt snippet you posted refers to upgrading from versions below
2.0 to 2.1
3) 2.0.1 is 2.0, not 2.1
Therefore the problem is actually
https://issues.apache.org/jira/browse/CASSANDRA-6093 ?
=Rob
RE: 1.2.10 -> 2.0.1 migration issue
Posted by Christopher Wirt <ch...@struq.com>.
Should also say. I have managed to move one node from 1.2.10 to 2.0.0. I'm
seeing this error on the machine I tried to migrate earlier to 2.0.1
Thanks
From: Christopher Wirt [mailto:chris.wirt@struq.com]
Sent: 25 September 2013 14:04
To: 'user@cassandra.apache.org'
Subject: RE: 1.2.10 -> 2.0.1 migration issue
Hi Marcus,
I've seen your patch. This works with what I'm seeing. The first data
directory only contained the JSON manifest at that time.
As a workaround I've made sure that each of the snapshot directories now
exist before starting up.
I still end up with the second exception I posted regarding a duplicate hard
link. Possibly two unrelated exceptions.
After getting this error. Looking at the datadirs
Data1 contains
JSON manifests
Loads of data files
Snapshot directory
Data2 contains
Just the snapshot directory
Data3 contains
Just the snapshot directory
INFO 12:56:22,766 Migrating manifest for struqrealtime/impressionstorev2
INFO 12:56:22,767 Snapshotting struqrealtime, impressionstorev2 to
pre-sstablemetamigration
ERROR 12:56:22,787 Exception encountered during startup
java.lang.RuntimeException: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
43)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
java.lang.RuntimeException: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
43)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
Exception encountered during startup: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json
Thanks,
Chris
From: Marcus Eriksson [mailto:krummas@gmail.com]
Sent: 25 September 2013 13:11
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue
you are probably reading trunk NEWS.txt
read the ticket for explanation of what the issue was (it is a proper bug)
On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt <ch...@struq.com>
wrote:
Hi Marcus,
Thanks for having a look at this.
Just noticed this in the NEWS.txt
For leveled compaction users, 2.0 must be atleast started before
upgrading to 2.1 due to the fact that the old JSON leveled
manifest is migrated into the sstable metadata files on startup
in 2.0 and this code is gone from 2.1.
Basically, my fault for skimming over this too quickly.
We will move from 1.2.10 -> 2.0 -> 2.1
Thanks,
Chris
From: Marcus Eriksson [mailto:krummas@gmail.com]
Sent: 25 September 2013 09:37
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue
cant really reproduce, could you update the ticket with a bit more info
about your setup?
do you have multiple .json files in your data dirs?
On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson <kr...@gmail.com> wrote:
this is most likely a bug, filed
https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a
look today.
On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt <ch...@struq.com>
wrote:
Hi,
Just had a go at upgrading a node to the latest stable c* 2 release and
think I ran into some issues with manifest migration.
On initial start up I hit this error as it starts to load the first of my
CF.
INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2
INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration
ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
Exception encountered during startup
FSWriteError in
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
42)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
Caused by: java.nio.file.NoSuchFileException:
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json ->
/disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474
)
at java.nio.file.Files.createLink(Files.java:1037)
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)
... 5 more
I had already successful run a test migration on our dev server. Only real
difference I can see if the number of data directories defined and the
amount of data being held.
I've run upgradesstables under 1.2.10. I have always been using vnodes and
CQL3. I recently moved to using LZ4 instead of Snappy..
I tried to startup again and it gave me a slightly different error
INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2
INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration
ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
Exception encountered during startup
java.lang.RuntimeException: Tried to create duplicate hard link to
/disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:129)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
42)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
Will have a go recreating this tomorrow.
Any insight or guesses at what the issue might be are always welcome.
Thanks,
Chris
Re: 1.2.10 -> 2.0.1 migration issue
Posted by Marcus Eriksson <kr...@gmail.com>.
you are probably reading trunk NEWS.txt
read the ticket for explanation of what the issue was (it is a proper bug)
On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt <ch...@struq.com>wrote:
> Hi Marcus,****
>
> Thanks for having a look at this.****
>
> ** **
>
> Just noticed this in the NEWS.txt ****
>
> ** **
>
> For *leveled* compaction users, 2.0 must be *atleast* started before****
>
> upgrading to 2.1 due to the fact that the old JSON *leveled*****
>
> manifest is migrated into the *sstable* *metadata* files on startup**
> **
>
> in 2.0 and this code is gone from 2.1.****
>
> ** **
>
> Basically, my fault for skimming over this too quickly. ****
>
> ** **
>
> We will move from 1.2.10 -> 2.0 -> 2.1****
>
> ** **
>
> Thanks,****
>
> Chris****
>
> ** **
>
> ** **
>
> *From:* Marcus Eriksson [mailto:krummas@gmail.com]
> *Sent:* 25 September 2013 09:37
> *To:* user@cassandra.apache.org
> *Subject:* Re: 1.2.10 -> 2.0.1 migration issue****
>
> ** **
>
> cant really reproduce, could you update the ticket with a bit more info
> about your setup?****
>
> ** **
>
> do you have multiple .json files in your data dirs?****
>
> ** **
>
> On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson <kr...@gmail.com>
> wrote:****
>
> this is most likely a bug, filed
> https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have
> a look today.****
>
> ** **
>
> On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt <ch...@struq.com>
> wrote:****
>
> Hi,****
>
> ****
>
> Just had a go at upgrading a node to the latest stable c* 2 release and
> think I ran into some issues with manifest migration.****
>
> ****
>
> On initial start up I hit this error as it starts to load the first of my
> CF. ****
>
> ****
>
> INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
> Migrating manifest for struqrealtime/impressionstorev2****
>
> INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration*
> ***
>
> ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
> Exception encountered during startup****
>
> FSWriteError in
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ****
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
> ****
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> ****
>
> Caused by: java.nio.file.NoSuchFileException:
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ->
> /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json
> ****
>
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)****
>
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)****
>
> at
> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474)
> ****
>
> at java.nio.file.Files.createLink(Files.java:1037)****
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)**
> **
>
> ... 5 more****
>
> ****
>
> I had already successful run a test migration on our dev server. Only real
> difference I can see if the number of data directories defined and the
> amount of data being held. ****
>
> ****
>
> I’ve run upgradesstables under 1.2.10. I have always been using vnodes and
> CQL3. I recently moved to using LZ4 instead of Snappy..****
>
> ****
>
> I tried to startup again and it gave me a slightly different error****
>
> ****
>
> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
> Migrating manifest for struqrealtime/impressionstorev2****
>
> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration*
> ***
>
> ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
> Exception encountered during startup****
>
> java.lang.RuntimeException: Tried to create duplicate hard link to
> /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt
> ****
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129)
> ****
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> ****
>
> ****
>
> Will have a go recreating this tomorrow.****
>
> ****
>
> Any insight or guesses at what the issue might be are always welcome.****
>
> ****
>
> Thanks,****
>
> Chris****
>
> ** **
>
> ** **
>
RE: 1.2.10 -> 2.0.1 migration issue
Posted by Christopher Wirt <ch...@struq.com>.
Hi Marcus,
Thanks for having a look at this.
Just noticed this in the NEWS.txt
For leveled compaction users, 2.0 must be atleast started before
upgrading to 2.1 due to the fact that the old JSON leveled
manifest is migrated into the sstable metadata files on startup
in 2.0 and this code is gone from 2.1.
Basically, my fault for skimming over this too quickly.
We will move from 1.2.10 -> 2.0 -> 2.1
Thanks,
Chris
From: Marcus Eriksson [mailto:krummas@gmail.com]
Sent: 25 September 2013 09:37
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue
cant really reproduce, could you update the ticket with a bit more info
about your setup?
do you have multiple .json files in your data dirs?
On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson <kr...@gmail.com> wrote:
this is most likely a bug, filed
https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a
look today.
On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt <ch...@struq.com>
wrote:
Hi,
Just had a go at upgrading a node to the latest stable c* 2 release and
think I ran into some issues with manifest migration.
On initial start up I hit this error as it starts to load the first of my
CF.
INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2
INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration
ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
Exception encountered during startup
FSWriteError in
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
42)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
Caused by: java.nio.file.NoSuchFileException:
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json ->
/disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474
)
at java.nio.file.Files.createLink(Files.java:1037)
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)
... 5 more
I had already successful run a test migration on our dev server. Only real
difference I can see if the number of data directories defined and the
amount of data being held.
I've run upgradesstables under 1.2.10. I have always been using vnodes and
CQL3. I recently moved to using LZ4 instead of Snappy..
I tried to startup again and it gave me a slightly different error
INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2
INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration
ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
Exception encountered during startup
java.lang.RuntimeException: Tried to create duplicate hard link to
/disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:129)
at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
42)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
Will have a go recreating this tomorrow.
Any insight or guesses at what the issue might be are always welcome.
Thanks,
Chris
Re: 1.2.10 -> 2.0.1 migration issue
Posted by Marcus Eriksson <kr...@gmail.com>.
cant really reproduce, could you update the ticket with a bit more info
about your setup?
do you have multiple .json files in your data dirs?
On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson <kr...@gmail.com> wrote:
> this is most likely a bug, filed
> https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have
> a look today.
>
>
> On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt <ch...@struq.com>wrote:
>
>> Hi,****
>>
>> ** **
>>
>> Just had a go at upgrading a node to the latest stable c* 2 release and
>> think I ran into some issues with manifest migration.****
>>
>> ** **
>>
>> On initial start up I hit this error as it starts to load the first of my
>> CF. ****
>>
>> ** **
>>
>> INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
>> Migrating manifest for struqrealtime/impressionstorev2****
>>
>> INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
>> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration
>> ****
>>
>> ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
>> Exception encountered during startup****
>>
>> FSWriteError in
>> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
>> ****
>>
>> at
>> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)*
>> ***
>>
>> at
>> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
>> ****
>>
>> at
>> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
>> ****
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
>> ****
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
>> ****
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
>> ****
>>
>> Caused by: java.nio.file.NoSuchFileException:
>> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
>> ->
>> /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json
>> ****
>>
>> at
>> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)***
>> *
>>
>> at
>> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)****
>>
>> at
>> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474)
>> ****
>>
>> at java.nio.file.Files.createLink(Files.java:1037)****
>>
>> at
>> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)*
>> ***
>>
>> ... 5 more****
>>
>> ** **
>>
>> I had already successful run a test migration on our dev server. Only
>> real difference I can see if the number of data directories defined and the
>> amount of data being held. ****
>>
>> ** **
>>
>> I’ve run upgradesstables under 1.2.10. I have always been using vnodes
>> and CQL3. I recently moved to using LZ4 instead of Snappy..****
>>
>> ** **
>>
>> I tried to startup again and it gave me a slightly different error****
>>
>> ** **
>>
>> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
>> Migrating manifest for struqrealtime/impressionstorev2****
>>
>> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
>> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration
>> ****
>>
>> ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
>> Exception encountered during startup****
>>
>> java.lang.RuntimeException: Tried to create duplicate hard link to
>> /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt
>> ****
>>
>> at
>> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)*
>> ***
>>
>> at
>> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129)
>> ****
>>
>> at
>> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
>> ****
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
>> ****
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
>> ****
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
>> ****
>>
>> ** **
>>
>> Will have a go recreating this tomorrow.****
>>
>> ** **
>>
>> Any insight or guesses at what the issue might be are always welcome.****
>>
>> ** **
>>
>> Thanks,****
>>
>> Chris****
>>
>
>
Re: 1.2.10 -> 2.0.1 migration issue
Posted by Marcus Eriksson <kr...@gmail.com>.
this is most likely a bug, filed
https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a
look today.
On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt <ch...@struq.com>wrote:
> Hi,****
>
> ** **
>
> Just had a go at upgrading a node to the latest stable c* 2 release and
> think I ran into some issues with manifest migration.****
>
> ** **
>
> On initial start up I hit this error as it starts to load the first of my
> CF. ****
>
> ** **
>
> INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
> Migrating manifest for struqrealtime/impressionstorev2****
>
> INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration*
> ***
>
> ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
> Exception encountered during startup****
>
> FSWriteError in
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ****
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
> ****
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> ****
>
> Caused by: java.nio.file.NoSuchFileException:
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ->
> /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json
> ****
>
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)****
>
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)****
>
> at
> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474)
> ****
>
> at java.nio.file.Files.createLink(Files.java:1037)****
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)**
> **
>
> ... 5 more****
>
> ** **
>
> I had already successful run a test migration on our dev server. Only real
> difference I can see if the number of data directories defined and the
> amount of data being held. ****
>
> ** **
>
> I’ve run upgradesstables under 1.2.10. I have always been using vnodes and
> CQL3. I recently moved to using LZ4 instead of Snappy..****
>
> ** **
>
> I tried to startup again and it gave me a slightly different error****
>
> ** **
>
> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
> Migrating manifest for struqrealtime/impressionstorev2****
>
> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration*
> ***
>
> ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
> Exception encountered during startup****
>
> java.lang.RuntimeException: Tried to create duplicate hard link to
> /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt
> ****
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129)
> ****
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> ****
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> ****
>
> ** **
>
> Will have a go recreating this tomorrow.****
>
> ** **
>
> Any insight or guesses at what the issue might be are always welcome.****
>
> ** **
>
> Thanks,****
>
> Chris****
>