You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Billie Rinaldi <bi...@apache.org> on 2016/04/08 16:40:21 UTC
Fwd: why compaction failure on one table brings other tables offline,
how to recover
*From:* Jayesh Patel
*Sent:* Thursday, April 07, 2016 4:36 PM
*To:* 'user@accumulo.apache.org' <us...@accumulo.apache.org>
*Subject:* RE: why compaction failure on one table brings other tables
offline, how to recover
I have a 3 node Accumulo 1.7 cluster with a few small tables (few MB in
size at most).
I had one of those table fail minc because I had configured a
SummingCombiner with FIXEDLEN but had smaller values:
MinC failed (trying to convert to long, but byte array isn't long enough,
wanted 8 found 1) to create
hdfs://instance-accumulo:8020/accumulo/tables/1/default_tablet/F0002bcs.rf_tmp
retrying ...
I have learned since to set the ‘lossy’ parameter to true to avoid this. *Why
is the default value for it false* if it can cause catastrophic failure
that you’ll read about ahead.
However, this brought other the tablets for other tables offline without
any apparent errors or warnings. *Can someone please explain why?*
In order to recover from this, I did a ‘droptable’ from the shell on the
affected tables, but they all got stuck in the ‘DELETING’ state. I was
able to finally delete them using zkcli ‘rmr’ command. *Is there a better
way?*
I’m assuming there is a more proper way because when I created the tables
again (with the same name), they went back to having a single offline
tablet right away. *Is this because there are “traces” of the old table
left behind that affect the new table even though the new table has a
different table id?* I ended up wiping out hdfs and recreating the
accumulo instance.
It seems that a small bug, writing 1 byte value instead of 8 bytes, caused
us to dump the whole accumulo instance. Luckily the data wasn’t that
important, but this whole episode makes us wonder why doing things the
right way (assuming there is a right way) wasn’t obvious or if Accumulo is
just very fragile.
Please ask away any questions/clarification you might have. We’ll
appreciate any input you might have so we make educated decisions about
using Accumulo going forward.
Thank you,
Jayesh
RE: Fwd: why compaction failure on one table brings other tables
offline, how to recover
Posted by Jayesh Patel <jp...@keywcorp.com>.
Well the total RAM on the VM is 6GB with no swap space, so the OS and other
Accumulo processes have enough. I meant that 300MB is currently available for
tserver process to use as reported by 'free'. tserver.sort.buffer.size is set
to 100MB. I was able to start it up today, some change in the dynamics I
guess.
-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com]
Sent: Tuesday, April 12, 2016 11:11 AM
To: user@accumulo.apache.org
Subject: Re: Fwd: why compaction failure on one table brings other tables
offline, how to recover
Jayesh Patel wrote:
> Josh, The OOM tserver process was killed by the kernel, it didn't hang
> around. I tried restarting it manually, but it ran out of memory
> right away and was killed again leaving the tablet offline. It must
> have a huge "recovery" log to go through. HDFS
> /accumulo/wal/instance-accumulo+9997/24e08581-a081-4b41-afc5-d75bdda6c
> f15 is about 42MB, and machine has about 300MB free and apparently not
> enough for tserver.
>
Ok, cool. If you're that constrained on resources, you can also try reducing
the property tserver.sort.buffer.size in accumulo-site.xml. It defaults to
200M, you could try 25M or 50M instead.
This is a buffer size that is used for sorting log edits during the recovery
process. This might help if you never make it through the recovery process.
300MB is a little low in general as far as headroom goes (especially when
you're already not giving Accumulo enough RAM). Typically, you want to ensure
that you give the operating system at least 1G of memory for itself.
Re: Fwd: why compaction failure on one table brings other tables
offline, how to recover
Posted by Josh Elser <jo...@gmail.com>.
Jayesh Patel wrote:
> Josh, The OOM tserver process was killed by the kernel, it didn't hang
> around. I tried restarting it manually, but it ran out of memory right
> away and was killed again leaving the tablet offline. It must have a
> huge "recovery" log to go through. HDFS
> /accumulo/wal/instance-accumulo+9997/24e08581-a081-4b41-afc5-d75bdda6cf15 is
> about 42MB, and machine has about 300MB free and apparently not enough
> for tserver.
>
Ok, cool. If you're that constrained on resources, you can also try
reducing the property tserver.sort.buffer.size in accumulo-site.xml. It
defaults to 200M, you could try 25M or 50M instead.
This is a buffer size that is used for sorting log edits during the
recovery process. This might help if you never make it through the
recovery process.
300MB is a little low in general as far as headroom goes (especially
when you're already not giving Accumulo enough RAM). Typically, you want
to ensure that you give the operating system at least 1G of memory for
itself.
RE: Fwd: why compaction failure on one table brings other tables
offline, how to recover
Posted by Jayesh Patel <jp...@keywcorp.com>.
ACCUMULO-4160 seems very relevant. Since all the tables/tablets didn't go offline in my case, it might be that it was able to load a few tables/tablets and then ran into the one it couldn't and that table and the tables after that remained offline. Does the loading order follow the order of table IDs?
Josh, The OOM tserver process was killed by the kernel, it didn't hang around. I tried restarting it manually, but it ran out of memory right away and was killed again leaving the tablet offline. It must have a huge "recovery" log to go through. HDFS /accumulo/wal/instance-accumulo+9997/24e08581-a081-4b41-afc5-d75bdda6cf15 is about 42MB, and machine has about 300MB free and apparently not enough for tserver.
Jayesh
-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com]
Sent: Monday, April 11, 2016 7:49 PM
To: user@accumulo.apache.org
Subject: Re: Fwd: why compaction failure on one table brings other tables offline, how to recover
Sorry, I meant that to Jayesh, not to you, Christopher :)
Christopher wrote:
> I just meant that if there is a problem loading one tablet, other
> tablets may stay indefinitely in an offline state due to
> ACCUMULO-4160, however it got to that point.
>
> On Mon, Apr 11, 2016 at 6:35 PM Josh Elser <josh.elser@gmail.com
> < <ma...@gmail.com> mailto:josh.elser@gmail.com>> wrote:
>
> Do you mean that after an OOME, the tserver process didn't die and got
> into this bad state with an permanently offline tablet?
>
> Christopher wrote:
> > You might be seeing
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ACCUMULO-2D4160&d=CwIDaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=yH4vdeYURv2hJ1Gmntk0uc08nV-pgsJRlR6tfCR4KBw&m=4GkK54V3yGOxqE3u6udXSNUxTDEoGB0GWOq0tAP0Isk&s=2HiWr-sQFmbnJRTKF9TMXKCtS4Wk-f3bgRBGnoKXZU4&e> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ACCUMULO-2D4160&d=CwIDaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=yH4vdeYURv2hJ1Gmntk0uc08nV-pgsJRlR6tfCR4KBw&m=4GkK54V3yGOxqE3u6udXSNUxTDEoGB0GWOq0tAP0Isk&s=2HiWr-sQFmbnJRTKF9TMXKCtS4Wk-f3bgRBGnoKXZU4&e=
> >
> > On Mon, Apr 11, 2016 at 5:52 PM Jayesh Patel <jpatel@keywcorp.com
> < <ma...@keywcorp.com> mailto:jpatel@keywcorp.com>
> > < <ma...@keywcorp.com> mailto:jpatel@keywcorp.com <ma...@keywcorp.com>>> wrote:
> >
> > There really aren't a lot of log messages that can explain why
> > tablets for other tables went offline except the following:
> >
> > 2016-04-11 13:32:18,258
> > [tserver.TabletServerResourceManager$AssignmentWatcher] WARN :
> > tserver:instance-accumulo-3 Assignment for 2<< has been
> running for
> > at least 973455566ms
> > java.lang.Exception: Assignment of 2<<
> > at sun.misc.Unsafe.park(Native Method)
> > at java.util.concurrent.locks.LockSupport.park(Unknown
> Source)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown
> > Source)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown
> > Source)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
> > at
> >
> java.util.concurrent.locks.ReentrantLock$FairSync.lock(Unknown Source)
> > at java.util.concurrent.locks.ReentrantLock.lock(Unknown
> Source)
> > at
> >
> org.apache.accumulo.tserver.TabletServer.acquireRecoveryMemory(TabletServer.java:2230)
> > at
> >
> org.apache.accumulo.tserver.TabletServer.access$2600(TabletServer.java:252)
> > at
> >
> org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2150)
> > at
> >
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> > at
> >
> org.apache.accumulo.tserver.ActiveAssignmentRunnable.run(ActiveAssignmentRunnable.java:61)
> > at
> >
> org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > Source)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > Source)
> > at
> >
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> > at java.lang.Thread.run(Unknown Source)
> >
> > Table 2<< here doesn't have the issue with minc failing and so
> > shouldn’t be offline. These messages happened on a restart of a
> > tserver if that offers any clues. All the nodes were rebooted at
> > that time due to a power failure. I'm assuming that it's tablet
> > went offline soon after this message first appeared in the logs.
> >
> > Other tidbit of note is that the Accumulo operates for hours/days
> > without taking the tablets offline even though minc is
> failing and
> > it's the crash of a tserver due to OutOfMemory situation in
> one case
> > that seems to have taken the tablet offline. Is it safe to
> assume
> > that other tservers are not able to pick up the tablets that are
> > failing minc from a crashed tserver?
> >
> > -----Original Message-----
> > From: Josh Elser [mailto:josh.elser@gmail.com
> < <ma...@gmail.com> mailto:josh.elser@gmail.com>
> > < <ma...@gmail.com> mailto:josh.elser@gmail.com <ma...@gmail.com>>]
> > Sent: Friday, April 08, 2016 10:52 AM
> > To: <ma...@accumulo.apache.org> user@accumulo.apache.org
> < <ma...@accumulo.apache.org> mailto:user@accumulo.apache.org> <mailto:user@accumulo.apache.org
> < <ma...@accumulo.apache.org> mailto:user@accumulo.apache.org>>
> > Subject: Re: Fwd: why compaction failure on one table brings
> other
> > tables offline, how to recover
> >
> >
> >
> > Billie Rinaldi wrote:
> > > *From:* Jayesh Patel
> > > *Sent:* Thursday, April 07, 2016 4:36 PM
> > > *To:* 'user@accumulo.apache.org
> < <ma...@accumulo.apache.org> mailto:user@accumulo.apache.org> <mailto:user@accumulo.apache.org
> < <ma...@accumulo.apache.org> mailto:user@accumulo.apache.org>>
> > <mailto:user@accumulo.apache.org
> < <ma...@accumulo.apache.org> mailto:user@accumulo.apache.org> <mailto:user@accumulo.apache.org
> < <ma...@accumulo.apache.org> mailto:user@accumulo.apache.org>>>'
> > > < <ma...@accumulo.apache.org> user@accumulo.apache.org <ma...@accumulo.apache.org>
> < <ma...@accumulo.apache.org> mailto:user@accumulo.apache.org <ma...@accumulo.apache.org>>
> > <mailto:user@accumulo.apache.org
> < <ma...@accumulo.apache.org> mailto:user@accumulo.apache.org> <mailto:user@accumulo.apache.org
> < <ma...@accumulo.apache.org> mailto:user@accumulo.apache.org>>>>
> > > *Subject:* RE: why compaction failure on one table brings other
> > tables
> > > offline, how to recover____
> > >
> > > __ __
> > >
> > > I have a 3 node Accumulo 1.7 cluster with a few small tables
> (few MB
> > > in size at most).____
> > >
> > > __ __
> > >
> > > I had one of those table fail minc because I had configured a
> > > SummingCombiner with FIXEDLEN but had smaller values:____
> > >
> > > MinC failed (trying to convert to long, but byte array isn't long
> > > enough, wanted 8 found 1) to create
> > >
> >
> hdfs://instance-accumulo:8020/accumulo/tables/1/default_tablet/F0002bc
> > > s.rf_tmp
> > > retrying ...____
> > >
> > > __ __
> > >
> > > I have learned since to set the ‘lossy’ parameter to true to
> > avoid this.
> > > *Why is the default value for it false* if it can cause
> catastrophic
> > > failure that you’ll read about ahead.____
> >
> > I'm pretty sure I told you this on StackOverflow, but if
> you're not
> > writing 8-byte long values, don't used FIXEDLEN. Use VARLEN
> instead.
> >
> > > However, this brought other the tablets for other tables offline
> > > without any apparent errors or warnings. *Can someone please
> explain
> > > why?*____
> >
> > Can you provide logs? We are not wizards :)
> >
> > > In order to recover from this, I did a ‘droptable’ from the
> shell on
> > > the affected tables, but they all got stuck in the ‘DELETING’
> state.
> > > I was able to finally delete them using zkcli ‘rmr’ command. *Is
> > there
> > > a better way?____*
> >
> > Again, not sure why they would have gotten stuck in the deleting
> > phase without more logs/context (nor how far along in the
> deletion
> > process they got). It's possible that there were still entries in
> > the accumulo.metadata table.
> >
> > > I’m assuming there is a more proper way because when I created the
> > > tables again (with the same name), they went back to having a
> single
> > > offline tablet right away. *Is this because there are “traces”
> of the
> > > old table left behind that affect the new table even though the new
> > > table has a different table id?* I ended up wiping out hdfs and
> > > recreating the accumulo instance. ____
> >
> > Accumulo uses monotonically increasing IDs to identify
> tables. The
> > human-readable names are only there for your benefit. Creating a
> > table with the same name would not cause a problem. It sounds
> like
> > you got the metadata table in a bad state or have
> tabletservers in a
> > bad state (if you haven't restarted them).
> >
> > > It seems that a small bug, writing 1 byte value instead of 8 bytes,
> > > caused us to dump the whole accumulo instance. Luckily the data
> > > wasn’t that important, but this whole episode makes us wonder why
> > > doing things the right way (assuming there is a right way) wasn’t
> > > obvious or if Accumulo is just very fragile.____
> > >
> >
> > Causing Accumulo to be unable to flush data from memory to
> disk in a
> > minor compaction is a very bad idea. One that we cannot
> > automatically recover from because of the combiner
> configuration you
> > set.
> >
> > If you can provide logs and stack traces from the Accumulo
> services,
> > we can try to help you further. This is not normal. If you don't
> > believe me, take a look at the distributed tests we run each
> release
> > where we write hundreds of gigabytes of data across many servers
> > while randomly killing Accumulo processes.
> >
> > >
> > > Please ask away any questions/clarification you might have. We’ll
> > > appreciate any input you might have so we make educated decisions
> > > about using Accumulo going forward.____
> > >
> > > __ __
> > >
> > > Thank you,____
> > >
> > > Jayesh____
> > >
> > >
> >
>
Re: Fwd: why compaction failure on one table brings other tables
offline, how to recover
Posted by Josh Elser <jo...@gmail.com>.
Sorry, I meant that to Jayesh, not to you, Christopher :)
Christopher wrote:
> I just meant that if there is a problem loading one tablet, other
> tablets may stay indefinitely in an offline state due to ACCUMULO-4160,
> however it got to that point.
>
> On Mon, Apr 11, 2016 at 6:35 PM Josh Elser <josh.elser@gmail.com
> <ma...@gmail.com>> wrote:
>
> Do you mean that after an OOME, the tserver process didn't die and got
> into this bad state with an permanently offline tablet?
>
> Christopher wrote:
> > You might be seeing
> https://issues.apache.org/jira/browse/ACCUMULO-4160
> >
> > On Mon, Apr 11, 2016 at 5:52 PM Jayesh Patel <jpatel@keywcorp.com
> <ma...@keywcorp.com>
> > <mailto:jpatel@keywcorp.com <ma...@keywcorp.com>>> wrote:
> >
> > There really aren't a lot of log messages that can explain why
> > tablets for other tables went offline except the following:
> >
> > 2016-04-11 13:32:18,258
> > [tserver.TabletServerResourceManager$AssignmentWatcher] WARN :
> > tserver:instance-accumulo-3 Assignment for 2<< has been
> running for
> > at least 973455566ms
> > java.lang.Exception: Assignment of 2<<
> > at sun.misc.Unsafe.park(Native Method)
> > at java.util.concurrent.locks.LockSupport.park(Unknown
> Source)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown
> > Source)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown
> > Source)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
> > at
> >
> java.util.concurrent.locks.ReentrantLock$FairSync.lock(Unknown Source)
> > at java.util.concurrent.locks.ReentrantLock.lock(Unknown
> Source)
> > at
> >
> org.apache.accumulo.tserver.TabletServer.acquireRecoveryMemory(TabletServer.java:2230)
> > at
> >
> org.apache.accumulo.tserver.TabletServer.access$2600(TabletServer.java:252)
> > at
> >
> org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2150)
> > at
> >
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> > at
> >
> org.apache.accumulo.tserver.ActiveAssignmentRunnable.run(ActiveAssignmentRunnable.java:61)
> > at
> >
> org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > Source)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > Source)
> > at
> >
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> > at java.lang.Thread.run(Unknown Source)
> >
> > Table 2<< here doesn't have the issue with minc failing and so
> > shouldn’t be offline. These messages happened on a restart of a
> > tserver if that offers any clues. All the nodes were rebooted at
> > that time due to a power failure. I'm assuming that it's tablet
> > went offline soon after this message first appeared in the logs.
> >
> > Other tidbit of note is that the Accumulo operates for hours/days
> > without taking the tablets offline even though minc is
> failing and
> > it's the crash of a tserver due to OutOfMemory situation in
> one case
> > that seems to have taken the tablet offline. Is it safe to
> assume
> > that other tservers are not able to pick up the tablets that are
> > failing minc from a crashed tserver?
> >
> > -----Original Message-----
> > From: Josh Elser [mailto:josh.elser@gmail.com
> <ma...@gmail.com>
> > <mailto:josh.elser@gmail.com <ma...@gmail.com>>]
> > Sent: Friday, April 08, 2016 10:52 AM
> > To: user@accumulo.apache.org
> <ma...@accumulo.apache.org> <mailto:user@accumulo.apache.org
> <ma...@accumulo.apache.org>>
> > Subject: Re: Fwd: why compaction failure on one table brings
> other
> > tables offline, how to recover
> >
> >
> >
> > Billie Rinaldi wrote:
> > > *From:* Jayesh Patel
> > > *Sent:* Thursday, April 07, 2016 4:36 PM
> > > *To:* 'user@accumulo.apache.org
> <ma...@accumulo.apache.org> <mailto:user@accumulo.apache.org
> <ma...@accumulo.apache.org>>
> > <mailto:user@accumulo.apache.org
> <ma...@accumulo.apache.org> <mailto:user@accumulo.apache.org
> <ma...@accumulo.apache.org>>>'
> > > <user@accumulo.apache.org <ma...@accumulo.apache.org>
> <mailto:user@accumulo.apache.org <ma...@accumulo.apache.org>>
> > <mailto:user@accumulo.apache.org
> <ma...@accumulo.apache.org> <mailto:user@accumulo.apache.org
> <ma...@accumulo.apache.org>>>>
> > > *Subject:* RE: why compaction failure on one table brings other
> > tables
> > > offline, how to recover____
> > >
> > > __ __
> > >
> > > I have a 3 node Accumulo 1.7 cluster with a few small tables
> (few MB
> > > in size at most).____
> > >
> > > __ __
> > >
> > > I had one of those table fail minc because I had configured a
> > > SummingCombiner with FIXEDLEN but had smaller values:____
> > >
> > > MinC failed (trying to convert to long, but byte array isn't long
> > > enough, wanted 8 found 1) to create
> > >
> >
> hdfs://instance-accumulo:8020/accumulo/tables/1/default_tablet/F0002bc
> > > s.rf_tmp
> > > retrying ...____
> > >
> > > __ __
> > >
> > > I have learned since to set the ‘lossy’ parameter to true to
> > avoid this.
> > > *Why is the default value for it false* if it can cause
> catastrophic
> > > failure that you’ll read about ahead.____
> >
> > I'm pretty sure I told you this on StackOverflow, but if
> you're not
> > writing 8-byte long values, don't used FIXEDLEN. Use VARLEN
> instead.
> >
> > > However, this brought other the tablets for other tables offline
> > > without any apparent errors or warnings. *Can someone please
> explain
> > > why?*____
> >
> > Can you provide logs? We are not wizards :)
> >
> > > In order to recover from this, I did a ‘droptable’ from the
> shell on
> > > the affected tables, but they all got stuck in the ‘DELETING’
> state.
> > > I was able to finally delete them using zkcli ‘rmr’ command. *Is
> > there
> > > a better way?____*
> >
> > Again, not sure why they would have gotten stuck in the deleting
> > phase without more logs/context (nor how far along in the
> deletion
> > process they got). It's possible that there were still entries in
> > the accumulo.metadata table.
> >
> > > I’m assuming there is a more proper way because when I created the
> > > tables again (with the same name), they went back to having a
> single
> > > offline tablet right away. *Is this because there are “traces”
> of the
> > > old table left behind that affect the new table even though the new
> > > table has a different table id?* I ended up wiping out hdfs and
> > > recreating the accumulo instance. ____
> >
> > Accumulo uses monotonically increasing IDs to identify
> tables. The
> > human-readable names are only there for your benefit. Creating a
> > table with the same name would not cause a problem. It sounds
> like
> > you got the metadata table in a bad state or have
> tabletservers in a
> > bad state (if you haven't restarted them).
> >
> > > It seems that a small bug, writing 1 byte value instead of 8 bytes,
> > > caused us to dump the whole accumulo instance. Luckily the data
> > > wasn’t that important, but this whole episode makes us wonder why
> > > doing things the right way (assuming there is a right way) wasn’t
> > > obvious or if Accumulo is just very fragile.____
> > >
> >
> > Causing Accumulo to be unable to flush data from memory to
> disk in a
> > minor compaction is a very bad idea. One that we cannot
> > automatically recover from because of the combiner
> configuration you
> > set.
> >
> > If you can provide logs and stack traces from the Accumulo
> services,
> > we can try to help you further. This is not normal. If you don't
> > believe me, take a look at the distributed tests we run each
> release
> > where we write hundreds of gigabytes of data across many servers
> > while randomly killing Accumulo processes.
> >
> > >
> > > Please ask away any questions/clarification you might have. We’ll
> > > appreciate any input you might have so we make educated decisions
> > > about using Accumulo going forward.____
> > >
> > > __ __
> > >
> > > Thank you,____
> > >
> > > Jayesh____
> > >
> > >
> >
>
Re: Fwd: why compaction failure on one table brings other tables
offline, how to recover
Posted by Christopher <ct...@apache.org>.
I just meant that if there is a problem loading one tablet, other tablets
may stay indefinitely in an offline state due to ACCUMULO-4160, however it
got to that point.
On Mon, Apr 11, 2016 at 6:35 PM Josh Elser <jo...@gmail.com> wrote:
> Do you mean that after an OOME, the tserver process didn't die and got
> into this bad state with an permanently offline tablet?
>
> Christopher wrote:
> > You might be seeing https://issues.apache.org/jira/browse/ACCUMULO-4160
> >
> > On Mon, Apr 11, 2016 at 5:52 PM Jayesh Patel <jpatel@keywcorp.com
> > <ma...@keywcorp.com>> wrote:
> >
> > There really aren't a lot of log messages that can explain why
> > tablets for other tables went offline except the following:
> >
> > 2016-04-11 13:32:18,258
> > [tserver.TabletServerResourceManager$AssignmentWatcher] WARN :
> > tserver:instance-accumulo-3 Assignment for 2<< has been running for
> > at least 973455566ms
> > java.lang.Exception: Assignment of 2<<
> > at sun.misc.Unsafe.park(Native Method)
> > at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown
> > Source)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown
> > Source)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown
> Source)
> > at
> > java.util.concurrent.locks.ReentrantLock$FairSync.lock(Unknown
> Source)
> > at java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
> > at
> >
> org.apache.accumulo.tserver.TabletServer.acquireRecoveryMemory(TabletServer.java:2230)
> > at
> >
> org.apache.accumulo.tserver.TabletServer.access$2600(TabletServer.java:252)
> > at
> >
> org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2150)
> > at
> >
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> > at
> >
> org.apache.accumulo.tserver.ActiveAssignmentRunnable.run(ActiveAssignmentRunnable.java:61)
> > at
> > org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > Source)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > Source)
> > at
> >
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> > at java.lang.Thread.run(Unknown Source)
> >
> > Table 2<< here doesn't have the issue with minc failing and so
> > shouldn’t be offline. These messages happened on a restart of a
> > tserver if that offers any clues. All the nodes were rebooted at
> > that time due to a power failure. I'm assuming that it's tablet
> > went offline soon after this message first appeared in the logs.
> >
> > Other tidbit of note is that the Accumulo operates for hours/days
> > without taking the tablets offline even though minc is failing and
> > it's the crash of a tserver due to OutOfMemory situation in one case
> > that seems to have taken the tablet offline. Is it safe to assume
> > that other tservers are not able to pick up the tablets that are
> > failing minc from a crashed tserver?
> >
> > -----Original Message-----
> > From: Josh Elser [mailto:josh.elser@gmail.com
> > <ma...@gmail.com>]
> > Sent: Friday, April 08, 2016 10:52 AM
> > To: user@accumulo.apache.org <ma...@accumulo.apache.org>
> > Subject: Re: Fwd: why compaction failure on one table brings other
> > tables offline, how to recover
> >
> >
> >
> > Billie Rinaldi wrote:
> > > *From:* Jayesh Patel
> > > *Sent:* Thursday, April 07, 2016 4:36 PM
> > > *To:* 'user@accumulo.apache.org <ma...@accumulo.apache.org>
> > <mailto:user@accumulo.apache.org <ma...@accumulo.apache.org>>'
> > > <user@accumulo.apache.org <ma...@accumulo.apache.org>
> > <mailto:user@accumulo.apache.org <ma...@accumulo.apache.org>>>
> > > *Subject:* RE: why compaction failure on one table brings other
> > tables
> > > offline, how to recover____
> > >
> > > __ __
> > >
> > > I have a 3 node Accumulo 1.7 cluster with a few small tables (few
> MB
> > > in size at most).____
> > >
> > > __ __
> > >
> > > I had one of those table fail minc because I had configured a
> > > SummingCombiner with FIXEDLEN but had smaller values:____
> > >
> > > MinC failed (trying to convert to long, but byte array isn't long
> > > enough, wanted 8 found 1) to create
> > >
> >
> hdfs://instance-accumulo:8020/accumulo/tables/1/default_tablet/F0002bc
> > > s.rf_tmp
> > > retrying ...____
> > >
> > > __ __
> > >
> > > I have learned since to set the ‘lossy’ parameter to true to
> > avoid this.
> > > *Why is the default value for it false* if it can cause
> catastrophic
> > > failure that you’ll read about ahead.____
> >
> > I'm pretty sure I told you this on StackOverflow, but if you're not
> > writing 8-byte long values, don't used FIXEDLEN. Use VARLEN instead.
> >
> > > However, this brought other the tablets for other tables offline
> > > without any apparent errors or warnings. *Can someone please
> explain
> > > why?*____
> >
> > Can you provide logs? We are not wizards :)
> >
> > > In order to recover from this, I did a ‘droptable’ from the shell
> on
> > > the affected tables, but they all got stuck in the ‘DELETING’
> state.
> > > I was able to finally delete them using zkcli ‘rmr’ command. *Is
> > there
> > > a better way?____*
> >
> > Again, not sure why they would have gotten stuck in the deleting
> > phase without more logs/context (nor how far along in the deletion
> > process they got). It's possible that there were still entries in
> > the accumulo.metadata table.
> >
> > > I’m assuming there is a more proper way because when I created the
> > > tables again (with the same name), they went back to having a
> single
> > > offline tablet right away. *Is this because there are “traces” of
> the
> > > old table left behind that affect the new table even though the
> new
> > > table has a different table id?* I ended up wiping out hdfs and
> > > recreating the accumulo instance. ____
> >
> > Accumulo uses monotonically increasing IDs to identify tables. The
> > human-readable names are only there for your benefit. Creating a
> > table with the same name would not cause a problem. It sounds like
> > you got the metadata table in a bad state or have tabletservers in a
> > bad state (if you haven't restarted them).
> >
> > > It seems that a small bug, writing 1 byte value instead of 8
> bytes,
> > > caused us to dump the whole accumulo instance. Luckily the data
> > > wasn’t that important, but this whole episode makes us wonder why
> > > doing things the right way (assuming there is a right way) wasn’t
> > > obvious or if Accumulo is just very fragile.____
> > >
> >
> > Causing Accumulo to be unable to flush data from memory to disk in a
> > minor compaction is a very bad idea. One that we cannot
> > automatically recover from because of the combiner configuration you
> > set.
> >
> > If you can provide logs and stack traces from the Accumulo services,
> > we can try to help you further. This is not normal. If you don't
> > believe me, take a look at the distributed tests we run each release
> > where we write hundreds of gigabytes of data across many servers
> > while randomly killing Accumulo processes.
> >
> > >
> > > Please ask away any questions/clarification you might have. We’ll
> > > appreciate any input you might have so we make educated decisions
> > > about using Accumulo going forward.____
> > >
> > > __ __
> > >
> > > Thank you,____
> > >
> > > Jayesh____
> > >
> > >
> >
>
Re: Fwd: why compaction failure on one table brings other tables
offline, how to recover
Posted by Josh Elser <jo...@gmail.com>.
Do you mean that after an OOME, the tserver process didn't die and got
into this bad state with an permanently offline tablet?
Christopher wrote:
> You might be seeing https://issues.apache.org/jira/browse/ACCUMULO-4160
>
> On Mon, Apr 11, 2016 at 5:52 PM Jayesh Patel <jpatel@keywcorp.com
> <ma...@keywcorp.com>> wrote:
>
> There really aren't a lot of log messages that can explain why
> tablets for other tables went offline except the following:
>
> 2016-04-11 13:32:18,258
> [tserver.TabletServerResourceManager$AssignmentWatcher] WARN :
> tserver:instance-accumulo-3 Assignment for 2<< has been running for
> at least 973455566ms
> java.lang.Exception: Assignment of 2<<
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown
> Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown
> Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
> at
> java.util.concurrent.locks.ReentrantLock$FairSync.lock(Unknown Source)
> at java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
> at
> org.apache.accumulo.tserver.TabletServer.acquireRecoveryMemory(TabletServer.java:2230)
> at
> org.apache.accumulo.tserver.TabletServer.access$2600(TabletServer.java:252)
> at
> org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2150)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at
> org.apache.accumulo.tserver.ActiveAssignmentRunnable.run(ActiveAssignmentRunnable.java:61)
> at
> org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Unknown Source)
>
> Table 2<< here doesn't have the issue with minc failing and so
> shouldn’t be offline. These messages happened on a restart of a
> tserver if that offers any clues. All the nodes were rebooted at
> that time due to a power failure. I'm assuming that it's tablet
> went offline soon after this message first appeared in the logs.
>
> Other tidbit of note is that the Accumulo operates for hours/days
> without taking the tablets offline even though minc is failing and
> it's the crash of a tserver due to OutOfMemory situation in one case
> that seems to have taken the tablet offline. Is it safe to assume
> that other tservers are not able to pick up the tablets that are
> failing minc from a crashed tserver?
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com
> <ma...@gmail.com>]
> Sent: Friday, April 08, 2016 10:52 AM
> To: user@accumulo.apache.org <ma...@accumulo.apache.org>
> Subject: Re: Fwd: why compaction failure on one table brings other
> tables offline, how to recover
>
>
>
> Billie Rinaldi wrote:
> > *From:* Jayesh Patel
> > *Sent:* Thursday, April 07, 2016 4:36 PM
> > *To:* 'user@accumulo.apache.org <ma...@accumulo.apache.org>
> <mailto:user@accumulo.apache.org <ma...@accumulo.apache.org>>'
> > <user@accumulo.apache.org <ma...@accumulo.apache.org>
> <mailto:user@accumulo.apache.org <ma...@accumulo.apache.org>>>
> > *Subject:* RE: why compaction failure on one table brings other
> tables
> > offline, how to recover____
> >
> > __ __
> >
> > I have a 3 node Accumulo 1.7 cluster with a few small tables (few MB
> > in size at most).____
> >
> > __ __
> >
> > I had one of those table fail minc because I had configured a
> > SummingCombiner with FIXEDLEN but had smaller values:____
> >
> > MinC failed (trying to convert to long, but byte array isn't long
> > enough, wanted 8 found 1) to create
> >
> hdfs://instance-accumulo:8020/accumulo/tables/1/default_tablet/F0002bc
> > s.rf_tmp
> > retrying ...____
> >
> > __ __
> >
> > I have learned since to set the ‘lossy’ parameter to true to
> avoid this.
> > *Why is the default value for it false* if it can cause catastrophic
> > failure that you’ll read about ahead.____
>
> I'm pretty sure I told you this on StackOverflow, but if you're not
> writing 8-byte long values, don't used FIXEDLEN. Use VARLEN instead.
>
> > However, this brought other the tablets for other tables offline
> > without any apparent errors or warnings. *Can someone please explain
> > why?*____
>
> Can you provide logs? We are not wizards :)
>
> > In order to recover from this, I did a ‘droptable’ from the shell on
> > the affected tables, but they all got stuck in the ‘DELETING’ state.
> > I was able to finally delete them using zkcli ‘rmr’ command. *Is
> there
> > a better way?____*
>
> Again, not sure why they would have gotten stuck in the deleting
> phase without more logs/context (nor how far along in the deletion
> process they got). It's possible that there were still entries in
> the accumulo.metadata table.
>
> > I’m assuming there is a more proper way because when I created the
> > tables again (with the same name), they went back to having a single
> > offline tablet right away. *Is this because there are “traces” of the
> > old table left behind that affect the new table even though the new
> > table has a different table id?* I ended up wiping out hdfs and
> > recreating the accumulo instance. ____
>
> Accumulo uses monotonically increasing IDs to identify tables. The
> human-readable names are only there for your benefit. Creating a
> table with the same name would not cause a problem. It sounds like
> you got the metadata table in a bad state or have tabletservers in a
> bad state (if you haven't restarted them).
>
> > It seems that a small bug, writing 1 byte value instead of 8 bytes,
> > caused us to dump the whole accumulo instance. Luckily the data
> > wasn’t that important, but this whole episode makes us wonder why
> > doing things the right way (assuming there is a right way) wasn’t
> > obvious or if Accumulo is just very fragile.____
> >
>
> Causing Accumulo to be unable to flush data from memory to disk in a
> minor compaction is a very bad idea. One that we cannot
> automatically recover from because of the combiner configuration you
> set.
>
> If you can provide logs and stack traces from the Accumulo services,
> we can try to help you further. This is not normal. If you don't
> believe me, take a look at the distributed tests we run each release
> where we write hundreds of gigabytes of data across many servers
> while randomly killing Accumulo processes.
>
> >
> > Please ask away any questions/clarification you might have. We’ll
> > appreciate any input you might have so we make educated decisions
> > about using Accumulo going forward.____
> >
> > __ __
> >
> > Thank you,____
> >
> > Jayesh____
> >
> >
>
Re: Fwd: why compaction failure on one table brings other tables
offline, how to recover
Posted by Christopher <ct...@apache.org>.
You might be seeing https://issues.apache.org/jira/browse/ACCUMULO-4160
On Mon, Apr 11, 2016 at 5:52 PM Jayesh Patel <jp...@keywcorp.com> wrote:
> There really aren't a lot of log messages that can explain why tablets for
> other tables went offline except the following:
>
> 2016-04-11 13:32:18,258
> [tserver.TabletServerResourceManager$AssignmentWatcher] WARN :
> tserver:instance-accumulo-3 Assignment for 2<< has been running for at
> least 973455566ms
> java.lang.Exception: Assignment of 2<<
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown
> Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown
> Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown
> Source)
> at java.util.concurrent.locks.ReentrantLock$FairSync.lock(Unknown
> Source)
> at java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
> at
> org.apache.accumulo.tserver.TabletServer.acquireRecoveryMemory(TabletServer.java:2230)
> at
> org.apache.accumulo.tserver.TabletServer.access$2600(TabletServer.java:252)
> at
> org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2150)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at
> org.apache.accumulo.tserver.ActiveAssignmentRunnable.run(ActiveAssignmentRunnable.java:61)
> at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Unknown Source)
>
> Table 2<< here doesn't have the issue with minc failing and so shouldn’t
> be offline. These messages happened on a restart of a tserver if that
> offers any clues. All the nodes were rebooted at that time due to a power
> failure. I'm assuming that it's tablet went offline soon after this
> message first appeared in the logs.
>
> Other tidbit of note is that the Accumulo operates for hours/days without
> taking the tablets offline even though minc is failing and it's the crash
> of a tserver due to OutOfMemory situation in one case that seems to have
> taken the tablet offline. Is it safe to assume that other tservers are not
> able to pick up the tablets that are failing minc from a crashed tserver?
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Friday, April 08, 2016 10:52 AM
> To: user@accumulo.apache.org
> Subject: Re: Fwd: why compaction failure on one table brings other tables
> offline, how to recover
>
>
>
> Billie Rinaldi wrote:
> > *From:* Jayesh Patel
> > *Sent:* Thursday, April 07, 2016 4:36 PM
> > *To:* 'user@accumulo.apache.org <ma...@accumulo.apache.org>'
> > <user@accumulo.apache.org <ma...@accumulo.apache.org>>
> > *Subject:* RE: why compaction failure on one table brings other tables
> > offline, how to recover____
> >
> > __ __
> >
> > I have a 3 node Accumulo 1.7 cluster with a few small tables (few MB
> > in size at most).____
> >
> > __ __
> >
> > I had one of those table fail minc because I had configured a
> > SummingCombiner with FIXEDLEN but had smaller values:____
> >
> > MinC failed (trying to convert to long, but byte array isn't long
> > enough, wanted 8 found 1) to create
> > hdfs://instance-accumulo:8020/accumulo/tables/1/default_tablet/F0002bc
> > s.rf_tmp
> > retrying ...____
> >
> > __ __
> >
> > I have learned since to set the ‘lossy’ parameter to true to avoid this.
> > *Why is the default value for it false* if it can cause catastrophic
> > failure that you’ll read about ahead.____
>
> I'm pretty sure I told you this on StackOverflow, but if you're not
> writing 8-byte long values, don't used FIXEDLEN. Use VARLEN instead.
>
> > However, this brought other the tablets for other tables offline
> > without any apparent errors or warnings. *Can someone please explain
> > why?*____
>
> Can you provide logs? We are not wizards :)
>
> > In order to recover from this, I did a ‘droptable’ from the shell on
> > the affected tables, but they all got stuck in the ‘DELETING’ state.
> > I was able to finally delete them using zkcli ‘rmr’ command. *Is there
> > a better way?____*
>
> Again, not sure why they would have gotten stuck in the deleting phase
> without more logs/context (nor how far along in the deletion process they
> got). It's possible that there were still entries in the accumulo.metadata
> table.
>
> > I’m assuming there is a more proper way because when I created the
> > tables again (with the same name), they went back to having a single
> > offline tablet right away. *Is this because there are “traces” of the
> > old table left behind that affect the new table even though the new
> > table has a different table id?* I ended up wiping out hdfs and
> > recreating the accumulo instance. ____
>
> Accumulo uses monotonically increasing IDs to identify tables. The
> human-readable names are only there for your benefit. Creating a table with
> the same name would not cause a problem. It sounds like you got the
> metadata table in a bad state or have tabletservers in a bad state (if you
> haven't restarted them).
>
> > It seems that a small bug, writing 1 byte value instead of 8 bytes,
> > caused us to dump the whole accumulo instance. Luckily the data
> > wasn’t that important, but this whole episode makes us wonder why
> > doing things the right way (assuming there is a right way) wasn’t
> > obvious or if Accumulo is just very fragile.____
> >
>
> Causing Accumulo to be unable to flush data from memory to disk in a minor
> compaction is a very bad idea. One that we cannot automatically recover
> from because of the combiner configuration you set.
>
> If you can provide logs and stack traces from the Accumulo services, we
> can try to help you further. This is not normal. If you don't believe me,
> take a look at the distributed tests we run each release where we write
> hundreds of gigabytes of data across many servers while randomly killing
> Accumulo processes.
>
> >
> > Please ask away any questions/clarification you might have. We’ll
> > appreciate any input you might have so we make educated decisions
> > about using Accumulo going forward.____
> >
> > __ __
> >
> > Thank you,____
> >
> > Jayesh____
> >
> >
>
RE: Fwd: why compaction failure on one table brings other tables
offline, how to recover
Posted by Jayesh Patel <jp...@keywcorp.com>.
There really aren't a lot of log messages that can explain why tablets for other tables went offline except the following:
2016-04-11 13:32:18,258 [tserver.TabletServerResourceManager$AssignmentWatcher] WARN : tserver:instance-accumulo-3 Assignment for 2<< has been running for at least 973455566ms
java.lang.Exception: Assignment of 2<<
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
at java.util.concurrent.locks.ReentrantLock$FairSync.lock(Unknown Source)
at java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
at org.apache.accumulo.tserver.TabletServer.acquireRecoveryMemory(TabletServer.java:2230)
at org.apache.accumulo.tserver.TabletServer.access$2600(TabletServer.java:252)
at org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2150)
at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at org.apache.accumulo.tserver.ActiveAssignmentRunnable.run(ActiveAssignmentRunnable.java:61)
at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Unknown Source)
Table 2<< here doesn't have the issue with minc failing and so shouldn’t be offline. These messages happened on a restart of a tserver if that offers any clues. All the nodes were rebooted at that time due to a power failure. I'm assuming that it's tablet went offline soon after this message first appeared in the logs.
Other tidbit of note is that the Accumulo operates for hours/days without taking the tablets offline even though minc is failing and it's the crash of a tserver due to OutOfMemory situation in one case that seems to have taken the tablet offline. Is it safe to assume that other tservers are not able to pick up the tablets that are failing minc from a crashed tserver?
-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com]
Sent: Friday, April 08, 2016 10:52 AM
To: user@accumulo.apache.org
Subject: Re: Fwd: why compaction failure on one table brings other tables offline, how to recover
Billie Rinaldi wrote:
> *From:* Jayesh Patel
> *Sent:* Thursday, April 07, 2016 4:36 PM
> *To:* 'user@accumulo.apache.org <ma...@accumulo.apache.org>'
> <user@accumulo.apache.org <ma...@accumulo.apache.org>>
> *Subject:* RE: why compaction failure on one table brings other tables
> offline, how to recover____
>
> __ __
>
> I have a 3 node Accumulo 1.7 cluster with a few small tables (few MB
> in size at most).____
>
> __ __
>
> I had one of those table fail minc because I had configured a
> SummingCombiner with FIXEDLEN but had smaller values:____
>
> MinC failed (trying to convert to long, but byte array isn't long
> enough, wanted 8 found 1) to create
> hdfs://instance-accumulo:8020/accumulo/tables/1/default_tablet/F0002bc
> s.rf_tmp
> retrying ...____
>
> __ __
>
> I have learned since to set the ‘lossy’ parameter to true to avoid this.
> *Why is the default value for it false* if it can cause catastrophic
> failure that you’ll read about ahead.____
I'm pretty sure I told you this on StackOverflow, but if you're not writing 8-byte long values, don't used FIXEDLEN. Use VARLEN instead.
> However, this brought other the tablets for other tables offline
> without any apparent errors or warnings. *Can someone please explain
> why?*____
Can you provide logs? We are not wizards :)
> In order to recover from this, I did a ‘droptable’ from the shell on
> the affected tables, but they all got stuck in the ‘DELETING’ state.
> I was able to finally delete them using zkcli ‘rmr’ command. *Is there
> a better way?____*
Again, not sure why they would have gotten stuck in the deleting phase without more logs/context (nor how far along in the deletion process they got). It's possible that there were still entries in the accumulo.metadata table.
> I’m assuming there is a more proper way because when I created the
> tables again (with the same name), they went back to having a single
> offline tablet right away. *Is this because there are “traces” of the
> old table left behind that affect the new table even though the new
> table has a different table id?* I ended up wiping out hdfs and
> recreating the accumulo instance. ____
Accumulo uses monotonically increasing IDs to identify tables. The human-readable names are only there for your benefit. Creating a table with the same name would not cause a problem. It sounds like you got the metadata table in a bad state or have tabletservers in a bad state (if you haven't restarted them).
> It seems that a small bug, writing 1 byte value instead of 8 bytes,
> caused us to dump the whole accumulo instance. Luckily the data
> wasn’t that important, but this whole episode makes us wonder why
> doing things the right way (assuming there is a right way) wasn’t
> obvious or if Accumulo is just very fragile.____
>
Causing Accumulo to be unable to flush data from memory to disk in a minor compaction is a very bad idea. One that we cannot automatically recover from because of the combiner configuration you set.
If you can provide logs and stack traces from the Accumulo services, we can try to help you further. This is not normal. If you don't believe me, take a look at the distributed tests we run each release where we write hundreds of gigabytes of data across many servers while randomly killing Accumulo processes.
>
> Please ask away any questions/clarification you might have. We’ll
> appreciate any input you might have so we make educated decisions
> about using Accumulo going forward.____
>
> __ __
>
> Thank you,____
>
> Jayesh____
>
>
Re: Fwd: why compaction failure on one table brings other tables
offline, how to recover
Posted by Josh Elser <jo...@gmail.com>.
Billie Rinaldi wrote:
> *From:* Jayesh Patel
> *Sent:* Thursday, April 07, 2016 4:36 PM
> *To:* 'user@accumulo.apache.org <ma...@accumulo.apache.org>'
> <user@accumulo.apache.org <ma...@accumulo.apache.org>>
> *Subject:* RE: why compaction failure on one table brings other tables
> offline, how to recover____
>
> __ __
>
> I have a 3 node Accumulo 1.7 cluster with a few small tables (few MB in
> size at most).____
>
> __ __
>
> I had one of those table fail minc because I had configured a
> SummingCombiner with FIXEDLEN but had smaller values:____
>
> MinC failed (trying to convert to long, but byte array isn't long
> enough, wanted 8 found 1) to create
> hdfs://instance-accumulo:8020/accumulo/tables/1/default_tablet/F0002bcs.rf_tmp
> retrying ...____
>
> __ __
>
> I have learned since to set the ‘lossy’ parameter to true to avoid this.
> *Why is the default value for it false* if it can cause catastrophic
> failure that you’ll read about ahead.____
I'm pretty sure I told you this on StackOverflow, but if you're not
writing 8-byte long values, don't used FIXEDLEN. Use VARLEN instead.
> However, this brought other the tablets for other tables offline without
> any apparent errors or warnings. *Can someone please explain why?*____
Can you provide logs? We are not wizards :)
> In order to recover from this, I did a ‘droptable’ from the shell on the
> affected tables, but they all got stuck in the ‘DELETING’ state. I was
> able to finally delete them using zkcli ‘rmr’ command. *Is there a
> better way?____*
Again, not sure why they would have gotten stuck in the deleting phase
without more logs/context (nor how far along in the deletion process
they got). It's possible that there were still entries in the
accumulo.metadata table.
> I’m assuming there is a more proper way because when I created the
> tables again (with the same name), they went back to having a single
> offline tablet right away. *Is this because there are “traces” of the
> old table left behind that affect the new table even though the new
> table has a different table id?* I ended up wiping out hdfs and
> recreating the accumulo instance. ____
Accumulo uses monotonically increasing IDs to identify tables. The
human-readable names are only there for your benefit. Creating a table
with the same name would not cause a problem. It sounds like you got the
metadata table in a bad state or have tabletservers in a bad state (if
you haven't restarted them).
> It seems that a small bug, writing 1 byte value instead of 8 bytes,
> caused us to dump the whole accumulo instance. Luckily the data wasn’t
> that important, but this whole episode makes us wonder why doing things
> the right way (assuming there is a right way) wasn’t obvious or if
> Accumulo is just very fragile.____
>
Causing Accumulo to be unable to flush data from memory to disk in a
minor compaction is a very bad idea. One that we cannot automatically
recover from because of the combiner configuration you set.
If you can provide logs and stack traces from the Accumulo services, we
can try to help you further. This is not normal. If you don't believe
me, take a look at the distributed tests we run each release where we
write hundreds of gigabytes of data across many servers while randomly
killing Accumulo processes.
>
> Please ask away any questions/clarification you might have. We’ll
> appreciate any input you might have so we make educated decisions about
> using Accumulo going forward.____
>
> __ __
>
> Thank you,____
>
> Jayesh____
>
>