Re: CorruptBlockException

[Robert Coli <rc...@eventbrite.com>]

On Sat, Jun 29, 2013 at 8:39 PM, Glenn Thompson <ga...@gmail.com> wrote:
> I'm Glenn Thompson and new to Cassandra.  I have been trying to figure out
> how to recover from a CorruptBlockException.
> ...
> One of my nodes must have a hardware problem.  Although I've been unable to
> find anything wrong via logs, smart, or mce.
> ...
> The repair, scrub, and decommission all produced Exceptions related to the
> same few corrupt files.

Hardware problem sounds relatively likely, especially if you have not
crashed your nodes. Only other thing I can think of is an issue with
the relationship of the compression library and the JVM. What JVM/JDK
are you using, and what compression method is in use on the Column
Family?

In general the actions you took were reasonable. Do you have the full
stack trace?

=Rob

Re: CorruptBlockException

[Glenn Thompson <ga...@gmail.com>]

Hi Rob,

It was hardware.  Memory.  I've been loading data since I originally
posted.  No exceptions so far.  I had some issues with OOMs when I first
started playing with cassandra.  I increased the amount RAM to the VM and
reduced the memtable size.  I'm guessing it's because I'm using I3s.  More
cores would most likely improve GC performance.

I put all the logs and my configs on my google drive.  The link is in the
original post.  I'm running 1.2.4.  There have been two releases since my
original download.  I'm going to attempt an upgrade soon.

I'm also considering using leveled compaction.  I just have two 750GB
drives per node.  I'd like to use more than 50% of the drives if I can.

Thanks,
Glenn


On Mon, Jul 1, 2013 at 11:08 AM, Robert Coli <rc...@eventbrite.com> wrote:

> On Sat, Jun 29, 2013 at 8:39 PM, Glenn Thompson <ga...@gmail.com>
> wrote:
> > I'm Glenn Thompson and new to Cassandra.  I have been trying to figure
> out
> > how to recover from a CorruptBlockException.
> > ...
> > One of my nodes must have a hardware problem.  Although I've been unable
> to
> > find anything wrong via logs, smart, or mce.
> > ...
> > The repair, scrub, and decommission all produced Exceptions related to
> the
> > same few corrupt files.
>
> Hardware problem sounds relatively likely, especially if you have not
> crashed your nodes. Only other thing I can think of is an issue with
> the relationship of the compression library and the JVM. What JVM/JDK
> are you using, and what compression method is in use on the Column
> Family?
>
> In general the actions you took were reasonable. Do you have the full
> stack trace?
>
> =Rob
>