You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Jeff Kubina <je...@gmail.com> on 2016/10/12 22:14:23 UTC
How can I remove or fix a hanging droptable in Accumulo?
How can I remove or fix a hanging droptable in Accumulo?
Re: How can I remove or fix a hanging droptable in Accumulo?
Posted by Michael Wall <mj...@gmail.com>.
Jeff,
This is the bug I ran into,
https://issues.apache.org/jira/browse/ACCUMULO-4143. Looks like it was
around migrations instead of splitting.
Mike
On Wed, Oct 12, 2016 at 8:27 PM, Michael Wall <mj...@gmail.com> wrote:
> Jeff,
>
> There was a race condition around splitting I think, but I can't find the
> issue right now. I caught a stacktrace at one point doing just as Ed
> suggests, then setting 'debug on' in the shell try the droptable again.
> After I found the issue, I turned off all ingest and waited for the master
> stop balancing and tables to stop splitting. My droptable was successfully
> issued after that (doing Ed's trick once again).
>
> I can look for that issue tomorrow or if you catch a stacktrace, post it
> here.
>
> Mike
>
> On Wed, Oct 12, 2016 at 7:57 PM, Ed Coleman <de...@etcoleman.com> wrote:
>
>> This is a brute force approach and should only done as a last resort.
>> Sometimes bouncing the master(s) helps – make sure all of stopped, not just
>> failed over to another instance., but it sounds like you have tried that.
>>
>>
>>
>> You may be able to force it by manually editing zookeeper nodes using
>> zkCli.sh You are editing zookeeper so make sure you have the correct table
>> id and path(s)
>>
>>
>>
>> Add permission to write in zookeeper using:
>>
>>
>>
>> zkCli> addauth digest accumulo:YOUR_SECRET
>>
>>
>>
>> Under /accumulo/INSTANCE_ID/tables/TABLE_ID/state (going from memory,
>> could be fate) you can use examine the state
>>
>>
>>
>> zkCli> ls /accumulo/INSTANCE_ID/tables/TABLE_ID/
>>
>>
>>
>> there should be mutlple zk nodes, one of which is state (or maybe fate?)
>>
>>
>>
>> zkCli> get /accumulo/INSTANCE_ID/tables/TABLE_ID/state
>>
>>
>>
>> – it should say DELETING.
>>
>>
>>
>> zkCli> set /accumulo/INSTANCE_ID/tables/TABLE_ID/state ONLINE
>>
>>
>>
>> This will force the table back online. After all tablets are accounted
>> for in the monitor you should be able to re-issue the delete command via
>> the accumulo shell.
>>
>>
>>
>> Ed Coleman
>>
>>
>>
>> *From:* Jeff Kubina [mailto:jeff.kubina@gmail.com]
>> *Sent:* Wednesday, October 12, 2016 7:38 PM
>> *To:* user@accumulo.apache.org
>> *Subject:* Re: How can I remove or fix a hanging droptable in Accumulo?
>>
>>
>>
>> Lots of bulk ingest assignments. It's been hung for days and even
>> returned after a reboot of the master and tservers.
>>
>>
>> --
>>
>> Jeff Kubina
>> 410-988-4436
>>
>>
>>
>>
>>
>> On Wed, Oct 12, 2016 at 6:28 PM, Michael Wall <mj...@gmail.com> wrote:
>>
>> Jeff,
>>
>> What is the master doing while the droptable is hung?
>>
>> Mike
>>
>>
>>
>> On Wed, Oct 12, 2016 at 6:14 PM, Jeff Kubina <je...@gmail.com>
>> wrote:
>>
>> How can I remove or fix a hanging droptable in Accumulo?
>>
>>
>>
>>
>>
>
>
Re: How can I remove or fix a hanging droptable in Accumulo?
Posted by Michael Wall <mj...@gmail.com>.
Jeff,
There was a race condition around splitting I think, but I can't find the
issue right now. I caught a stacktrace at one point doing just as Ed
suggests, then setting 'debug on' in the shell try the droptable again.
After I found the issue, I turned off all ingest and waited for the master
stop balancing and tables to stop splitting. My droptable was successfully
issued after that (doing Ed's trick once again).
I can look for that issue tomorrow or if you catch a stacktrace, post it
here.
Mike
On Wed, Oct 12, 2016 at 7:57 PM, Ed Coleman <de...@etcoleman.com> wrote:
> This is a brute force approach and should only done as a last resort.
> Sometimes bouncing the master(s) helps – make sure all of stopped, not just
> failed over to another instance., but it sounds like you have tried that.
>
>
>
> You may be able to force it by manually editing zookeeper nodes using
> zkCli.sh You are editing zookeeper so make sure you have the correct table
> id and path(s)
>
>
>
> Add permission to write in zookeeper using:
>
>
>
> zkCli> addauth digest accumulo:YOUR_SECRET
>
>
>
> Under /accumulo/INSTANCE_ID/tables/TABLE_ID/state (going from memory,
> could be fate) you can use examine the state
>
>
>
> zkCli> ls /accumulo/INSTANCE_ID/tables/TABLE_ID/
>
>
>
> there should be mutlple zk nodes, one of which is state (or maybe fate?)
>
>
>
> zkCli> get /accumulo/INSTANCE_ID/tables/TABLE_ID/state
>
>
>
> – it should say DELETING.
>
>
>
> zkCli> set /accumulo/INSTANCE_ID/tables/TABLE_ID/state ONLINE
>
>
>
> This will force the table back online. After all tablets are accounted for
> in the monitor you should be able to re-issue the delete command via the
> accumulo shell.
>
>
>
> Ed Coleman
>
>
>
> *From:* Jeff Kubina [mailto:jeff.kubina@gmail.com]
> *Sent:* Wednesday, October 12, 2016 7:38 PM
> *To:* user@accumulo.apache.org
> *Subject:* Re: How can I remove or fix a hanging droptable in Accumulo?
>
>
>
> Lots of bulk ingest assignments. It's been hung for days and even returned
> after a reboot of the master and tservers.
>
>
> --
>
> Jeff Kubina
> 410-988-4436
>
>
>
>
>
> On Wed, Oct 12, 2016 at 6:28 PM, Michael Wall <mj...@gmail.com> wrote:
>
> Jeff,
>
> What is the master doing while the droptable is hung?
>
> Mike
>
>
>
> On Wed, Oct 12, 2016 at 6:14 PM, Jeff Kubina <je...@gmail.com>
> wrote:
>
> How can I remove or fix a hanging droptable in Accumulo?
>
>
>
>
>
RE: How can I remove or fix a hanging droptable in Accumulo?
Posted by Ed Coleman <de...@etcoleman.com>.
This is a brute force approach and should only done as a last resort. Sometimes bouncing the master(s) helps – make sure all of stopped, not just failed over to another instance., but it sounds like you have tried that.
You may be able to force it by manually editing zookeeper nodes using zkCli.sh You are editing zookeeper so make sure you have the correct table id and path(s)
Add permission to write in zookeeper using:
zkCli> addauth digest accumulo:YOUR_SECRET
Under /accumulo/INSTANCE_ID/tables/TABLE_ID/state (going from memory, could be fate) you can use examine the state
zkCli> ls /accumulo/INSTANCE_ID/tables/TABLE_ID/
there should be mutlple zk nodes, one of which is state (or maybe fate?)
zkCli> get /accumulo/INSTANCE_ID/tables/TABLE_ID/state
– it should say DELETING.
zkCli> set /accumulo/INSTANCE_ID/tables/TABLE_ID/state ONLINE
This will force the table back online. After all tablets are accounted for in the monitor you should be able to re-issue the delete command via the accumulo shell.
Ed Coleman
From: Jeff Kubina [mailto:jeff.kubina@gmail.com]
Sent: Wednesday, October 12, 2016 7:38 PM
To: user@accumulo.apache.org
Subject: Re: How can I remove or fix a hanging droptable in Accumulo?
Lots of bulk ingest assignments. It's been hung for days and even returned after a reboot of the master and tservers.
--
Jeff Kubina
410-988-4436
On Wed, Oct 12, 2016 at 6:28 PM, Michael Wall <mjwall@gmail.com <ma...@gmail.com> > wrote:
Jeff,
What is the master doing while the droptable is hung?
Mike
On Wed, Oct 12, 2016 at 6:14 PM, Jeff Kubina <jeff.kubina@gmail.com <ma...@gmail.com> > wrote:
How can I remove or fix a hanging droptable in Accumulo?
Re: How can I remove or fix a hanging droptable in Accumulo?
Posted by Jeff Kubina <je...@gmail.com>.
Lots of bulk ingest assignments. It's been hung for days and even returned
after a reboot of the master and tservers.
--
Jeff Kubina
410-988-4436
On Wed, Oct 12, 2016 at 6:28 PM, Michael Wall <mj...@gmail.com> wrote:
> Jeff,
>
> What is the master doing while the droptable is hung?
>
> Mike
>
> On Wed, Oct 12, 2016 at 6:14 PM, Jeff Kubina <je...@gmail.com>
> wrote:
>
>> How can I remove or fix a hanging droptable in Accumulo?
>>
>>
>
Re: How can I remove or fix a hanging droptable in Accumulo?
Posted by Michael Wall <mj...@gmail.com>.
Jeff,
What is the master doing while the droptable is hung?
Mike
On Wed, Oct 12, 2016 at 6:14 PM, Jeff Kubina <je...@gmail.com> wrote:
> How can I remove or fix a hanging droptable in Accumulo?
>
>