disk full error while bootstrapping

[Gurpreet Singh <gu...@gmail.com>]

Hi,
version: cassandra 0.6.5

I am trying to bootstrap a new node from an existing seed node.
The new node seems to be stuck with the bootstrapping message, and did not
show any activity.
Only after i checked the logs of the seed node, i realise there has been an
error:

Caused by: java.lang.UnsupportedOperationException: disk full
at
org.apache.cassandra.db.CompactionManager.doAntiCompaction(CompactionManager.java:345)
at
org.apache.cassandra.db.CompactionManager.access$500(CompactionManager.java:49)
at
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:143)
at
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:140)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)

The data size on the seed node is about 140 gigs, and it has abt 20 gig free
space left.
How do i get rid of this error and get the bootstrapping going ?
Can it anti-compact to some other place that has disk space available or any
other ways to get this working?

Thanks
Gurpreet

Re: disk full error while bootstrapping

[Gurpreet Singh <gu...@gmail.com>]

I did set autobootstrap true. It got the new token, and even proceeded to
print the message that its bootstrapping, however the source node just didnt
show any activity. At a later point, when i tried again (after the other
bootstrap from the other source was finished), it did proceed, however that
failed with a different exception during streaming. Anyway, getting rid of
that node now, as it has less memory (4  gigs) and giving streaming
exceptions, so just scping data out right now.

Thanks
Gurpreet

On Thu, Sep 9, 2010 at 9:09 PM, Jonathan Ellis <jb...@gmail.com> wrote:

> On Thu, Sep 9, 2010 at 2:24 PM, Gurpreet Singh <gu...@gmail.com>
> wrote:
> > D was once a part of the cluster, but had gone down because of disk
> issues.
> > Its back up, it still has the old data, however to bootstrap again, i
> > deleted the old Location db (is that a good practise?), and so i see it
> did
> > take a new token, but is not proceeding with the bootstrap.
>
> sounds like you didn't set autobootstrap=true.
>
> > Also, a question about the storage config file. When I brought up B, the
> > seeds configured in B's config was just A. When I am bringing up C and D,
> > the seeds configured in their config file are both A and B. B's config
> still
> > says the seed is A. For B to be a seed, does its own config need to also
> say
> > that its a seed.
>
> no.  but best practice is for all nodes to have the same seeds list to
> avoid confusion.
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Re: disk full error while bootstrapping

[Jonathan Ellis <jb...@gmail.com>]

On Thu, Sep 9, 2010 at 2:24 PM, Gurpreet Singh <gu...@gmail.com> wrote:
> D was once a part of the cluster, but had gone down because of disk issues.
> Its back up, it still has the old data, however to bootstrap again, i
> deleted the old Location db (is that a good practise?), and so i see it did
> take a new token, but is not proceeding with the bootstrap.

sounds like you didn't set autobootstrap=true.

> Also, a question about the storage config file. When I brought up B, the
> seeds configured in B's config was just A. When I am bringing up C and D,
> the seeds configured in their config file are both A and B. B's config still
> says the seed is A. For B to be a seed, does its own config need to also say
> that its a seed.

no.  but best practice is for all nodes to have the same seeds list to
avoid confusion.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: disk full error while bootstrapping

[Gurpreet Singh <gu...@gmail.com>]

Thanks Jonathan. I guess i need to be patient for JVM GC :-)

Two more things i was trying, and wanted to check if it was supported.
Now, i have a 2 node cluster (say A and B), and i am trying to bootstrap 2
more nodes. (C and D)
The first bootstrap started successfully. I see anticompaction happenning on
C for the purpose, and hopefully streaming shall start soon from C -> A
The second bootstrap I started only after the first one said bootstrapping
(as mentioned on the wiki). However, it just sits there, doing nothing (or
so I think! :-)).  D machine says its bootstrapping from B, and is supposed
to assume load from B. B just sits there, doing nothing.. unlike the first
case where I could see A machine logging messages about anticompaction.

D was once a part of the cluster, but had gone down because of disk issues.
Its back up, it still has the old data, however to bootstrap again, i
deleted the old Location db (is that a good practise?), and so i see it did
take a new token, but is not proceeding with the bootstrap. I wanted to see,
if bootstrap goes correctly in such cases, and testing nodetool cleanup
after that, and the amount of time all these things take.

Also, a question about the storage config file. When I brought up B, the
seeds configured in B's config was just A. When I am bringing up C and D,
the seeds configured in their config file are both A and B. B's config still
says the seed is A. For B to be a seed, does its own config need to also say
that its a seed.

Thanks for all the help,
Gurpreet

On Thu, Sep 9, 2010 at 7:25 AM, Jonathan Ellis <jb...@gmail.com> wrote:

> On Thu, Sep 9, 2010 at 12:50 AM, Gurpreet Singh
> <gu...@gmail.com> wrote:
> > 1. what is the purpose of this anticompacted file created during cleanup?
>
> That is all the data that still belongs to the node, post-bootstrap.
> Since you were just bringing the cluster back up to RF nodes, that's
> all the data it started with still.  The older files will be removed
> after the next JVM GC (or server restart, whichever comes first):
> http://wiki.apache.org/cassandra/MemtableSSTable
>
> > 2. compaction is also supposed to create a big file after reconciling the
> > sstable files.. i see that anti-compaction is also creating a big file.
> How
> > are these files going to be different?
>
> Anticompaction is for splitting out one range of data from the rest.
>
> > 3. Other than during streaming (bootstrapping), when is anti-compaction
> > triggered?
>
> It's not.
>
> > 4. Now i have 240 gigs of disk used for probably around 92 gigs of useful
> > data. When will the other files get deleted? Do i need to run compaction
> > separately to take care of this.
>
> Answered above.
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Re: disk full error while bootstrapping

[Jonathan Ellis <jb...@gmail.com>]

On Thu, Sep 9, 2010 at 12:50 AM, Gurpreet Singh
<gu...@gmail.com> wrote:
> 1. what is the purpose of this anticompacted file created during cleanup?

That is all the data that still belongs to the node, post-bootstrap.
Since you were just bringing the cluster back up to RF nodes, that's
all the data it started with still.  The older files will be removed
after the next JVM GC (or server restart, whichever comes first):
http://wiki.apache.org/cassandra/MemtableSSTable

> 2. compaction is also supposed to create a big file after reconciling the
> sstable files.. i see that anti-compaction is also creating a big file. How
> are these files going to be different?

Anticompaction is for splitting out one range of data from the rest.

> 3. Other than during streaming (bootstrapping), when is anti-compaction
> triggered?

It's not.

> 4. Now i have 240 gigs of disk used for probably around 92 gigs of useful
> data. When will the other files get deleted? Do i need to run compaction
> separately to take care of this.

Answered above.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: disk full error while bootstrapping

[Gurpreet Singh <gu...@gmail.com>]

Thanks Jonathan. I added an extra directory big enough and at least the
bootstrap worked.

However, it brings me to questions that i cant seem to find an answer to.

In my case, i had a 2 node cluster to start with, replication factor of 2,
basically all data residing on both machines. Both nodes had about 150 gigs
of data, and about 30 gigs free space.
Suddenly, one of the machines had a disk failure, so i decided to remove the
node, using nodetool removetoken. Brought up another node to replace the
down node, and tried to bootstrap it. Thats when I ran into the space issue.
Resolved it by adding the extra disk, about 100 gigs.
While bootstrapping I noticed anticompaction happenning for each
columnfamily, and saw the createion of a 92 gig file for the biggest
columnfamily i have, which was then streamed across to the other node. This
file was then deleted.  I followed this up with a nodetool cleanup, as was
mentioned on the wiki to delete any unused references (probably none in my
case, but just wanted to do it!). This triggered an anti-compaction again,
took 2 more hours and created a 92 gig file similar to the last step.
However, nothing happenned after this. This big file just sits there though,
and no other files were deleted. So, now i have 240 gigs of disk used up on
the node where i triggered the cleanup as no files were deleted. When I do a
nodetool ring, it says 93 gigs load, as opposed to 150 gigs previously.

Questions;
1. what is the purpose of this anticompacted file created during cleanup?
2. compaction is also supposed to create a big file after reconciling the
sstable files.. i see that anti-compaction is also creating a big file. How
are these files going to be different?
3. Other than during streaming (bootstrapping), when is anti-compaction
triggered?
4. Now i have 240 gigs of disk used for probably around 92 gigs of useful
data. When will the other files get deleted? Do i need to run compaction
separately to take care of this.

Thanks for your patience,
Gurpreet



On Wed, Sep 8, 2010 at 9:02 AM, Jonathan Ellis <jb...@gmail.com> wrote:

> if you have a data directory defined with enough room, Cassandra will
> use that one.
>
> On Wed, Sep 8, 2010 at 6:25 AM, Gurpreet Singh <gu...@gmail.com>
> wrote:
> > Hi,
> > version: cassandra 0.6.5
> > I am trying to bootstrap a new node from an existing seed node.
> > The new node seems to be stuck with the bootstrapping message, and did
> not
> > show any activity.
> > Only after i checked the logs of the seed node, i realise there has been
> an
> > error:
> > Caused by: java.lang.UnsupportedOperationException: disk full
> > at
> >
> org.apache.cassandra.db.CompactionManager.doAntiCompaction(CompactionManager.java:345)
> > at
> >
> org.apache.cassandra.db.CompactionManager.access$500(CompactionManager.java:49)
> > at
> >
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:143)
> > at
> >
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:140)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > The data size on the seed node is about 140 gigs, and it has abt 20 gig
> free
> > space left.
> > How do i get rid of this error and get the bootstrapping going ?
> > Can it anti-compact to some other place that has disk space available or
> any
> > other ways to get this working?
> > Thanks
> > Gurpreet
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Re: disk full error while bootstrapping

[Jonathan Ellis <jb...@gmail.com>]

if you have a data directory defined with enough room, Cassandra will
use that one.

On Wed, Sep 8, 2010 at 6:25 AM, Gurpreet Singh <gu...@gmail.com> wrote:
> Hi,
> version: cassandra 0.6.5
> I am trying to bootstrap a new node from an existing seed node.
> The new node seems to be stuck with the bootstrapping message, and did not
> show any activity.
> Only after i checked the logs of the seed node, i realise there has been an
> error:
> Caused by: java.lang.UnsupportedOperationException: disk full
> at
> org.apache.cassandra.db.CompactionManager.doAntiCompaction(CompactionManager.java:345)
> at
> org.apache.cassandra.db.CompactionManager.access$500(CompactionManager.java:49)
> at
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:143)
> at
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:140)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> The data size on the seed node is about 140 gigs, and it has abt 20 gig free
> space left.
> How do i get rid of this error and get the bootstrapping going ?
> Can it anti-compact to some other place that has disk space available or any
> other ways to get this working?
> Thanks
> Gurpreet



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com