You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Russ Weeks <rw...@newbrightidea.com> on 2014/02/27 05:10:28 UTC
Seeing lots of "Internal error processing waitForTableOperation"
Hi, All,
I can't delete any tables through the accumulo shell, I keep getting the
message: "Internal error processing waitForTableOperation".
Recently my accumulo cluster has started showing this error repeatedly in
the master log:
[state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< : []
9223372036854775807 false=loc243f8e8ad6f5875>.10.128.1.17:9997
~tab~pr-��}
And this error in the monitor log:
[util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet !0;!0<
has no prev end row
And this in the GC log:
[util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no prev
end row
[util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
9223372036854775807 false,~ : [] 9223372036854775807 false)
Maybe coincidence, but it seems like things started getting weird when I
added a WholeRowIterator through the accumulo shell (which is nonsense, I
know). The iterators on one of my tables (36M entries) look like this:
- Iterator WholeRowIterator, scan scope options:
- iteratorPriority = 50
- iteratorClassName =
org.apache.accumulo.core.iterators.user.WholeRowIterator
- Iterator rowdel, majc scope options:
- iteratorPriority = 30
- iteratorClassName =
org.apache.accumulo.core.iterators.user.RowDeletingIterator
- Iterator transformer, majc scope options:
- iteratorPriority = 15
- iteratorClassName =
com.phemi.agile.datapolicy.DataColumnTransformer
- maxBufferSize = 256M
- Iterator vers, majc scope options:
- iteratorPriority = 20
- iteratorClassName =
org.apache.accumulo.core.iterators.user.VersioningIterator
- maxVersions = 1
- Iterator vers, minc scope options:
- iteratorPriority = 20
- iteratorClassName =
org.apache.accumulo.core.iterators.user.VersioningIterator
- maxVersions = 1
- Iterator vers, scan scope options:
- iteratorPriority = 20
- iteratorClassName =
org.apache.accumulo.core.iterators.user.VersioningIterator
- maxVersions = 1
I tried deleting DataColumnTransformer, RowDeletingIterator and
WholeRowIterator. No errors were returned from the deleteiter command in
the shell but after running listiter, the WholeRowIterator and
RowDeletingIterator are still there.
I *am* still able to create tables, but tables that I create can't then be
deleted.
Any advice appreciated,
-Russ
Re: Seeing lots of "Internal error processing waitForTableOperation"
Posted by Russ Weeks <rw...@newbrightidea.com>.
Hi, Keith,
hduser@agile01:~/accumulo/bin$ accumulo
org.apache.accumulo.server.fate.Admin print
txid: 254bf92a31b14e3b status: SUCCESSFUL op: TableRangeOp
locked: [] locking: [] top: null
txid: 32cedb4ad5b9b3e3 status: SUCCESSFUL op: DeleteTable
locked: [] locking: [] top: null
I might have jumped the gun a bit, but I found how to use the zookeeper CLI
to force the table state to ONLINE... I can delete my tables now, I hope I
haven't messed up the FSM at a higher level.
-Russ
On Thu, Feb 27, 2014 at 2:00 PM, Keith Turner <ke...@deenlo.com> wrote:
> If you run 'accumulo org.apache.accumulo.server.fate.Admin -<list option,
> not sure what it is>', what does that look like?
>
>
>
>
> On Thu, Feb 27, 2014 at 4:50 PM, Russ Weeks <rw...@newbrightidea.com>wrote:
>
>> Looking at Zookeeper, it seems like some of my tables are stuck in the
>> DELETING state. I can't delete these tables. I *am* able to create and
>> delete new tables, or delete other tables that are not in the DELETING
>> state.
>>
>> Is there any way to force the table out of the DELETING state?
>>
>> Thanks,
>> -Russ
>>
>> [zk: localhost:2181(CONNECTED) 21] get
>> /accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/state
>> DELETING
>> cZxid = 0x2001138c0
>> ctime = Wed Feb 26 20:08:17 PST 2014
>> mZxid = 0x2001138ef
>> mtime = Wed Feb 26 20:08:24 PST 2014
>> pZxid = 0x2001138c0
>> cversion = 0
>> dataVersion = 2
>> aclVersion = 0
>> ephemeralOwner = 0x0
>> dataLength = 8
>> numChildren = 0
>> [zk: localhost:2181(CONNECTED) 22] get
>> /accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/name
>> test_table
>> cZxid = 0x2001138bf
>> ctime = Wed Feb 26 20:08:17 PST 2014
>> mZxid = 0x2001138bf
>> mtime = Wed Feb 26 20:08:17 PST 2014
>> pZxid = 0x2001138bf
>> cversion = 0
>> dataVersion = 0
>> aclVersion = 0
>> ephemeralOwner = 0x0
>> dataLength = 10
>> numChildren = 0
>>
>>
>>
>>
>> On Thu, Feb 27, 2014 at 1:34 PM, Russ Weeks <rw...@newbrightidea.com>wrote:
>>
>>> Hi, Keith,
>>>
>>> Thanks for your response. I think that's exactly what happened. I
>>> screwed up the config command and wound up specifying a couple of iterators
>>> on the !METADATA table (and indeed all tables in the system).
>>>
>>> I've removed those iterators using "config -d" and now I can scan the
>>> !METADATA table without errors in the shell or the master log. But I'm
>>> still not able to delete tables. When I run the command "deletetable
>>> test_table", I see this message in the shell:
>>>
>>> ERROR: org.apache.accumulo.core.client.AccumuloException: Internal error
>>> processing waitForTableOperation
>>>
>>> And this exception in the master log:
>>>
>>> 2014-02-27 13:32:12,311 [zookeeper.DistributedReadWriteLock] INFO :
>>> Added lock entry 0 userData 1a33c6891b997e71 lockType WRITE
>>> 2014-02-27 13:32:12,313 [tableOps.Utils] INFO : table 3n
>>> (1a33c6891b997e71) locked for write operation: DELETE
>>> 2014-02-27 13:32:12,314 [tables.TableManager] FATAL: Failed to
>>> transition table to state DELETING
>>> 2014-02-27 13:32:12,320 [fate.Fate] WARN : Failed to execute Repo,
>>> tid=1a33c6891b997e71
>>> java.lang.RuntimeException:
>>> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
>>> at
>>> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:148)
>>> at
>>> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:224)
>>> at
>>> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:207)
>>> at
>>> org.apache.accumulo.server.master.tableOps.TraceRepo.call(TraceRepo.java:65)
>>> at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:64)
>>> at
>>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34)
>>> at java.lang.Thread.run(Thread.java:724)
>>> Caused by:
>>> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
>>> at
>>> org.apache.accumulo.server.master.state.tables.TableManager$1.mutate(TableManager.java:141)
>>> at
>>> org.apache.accumulo.fate.zookeeper.ZooReaderWriter.mutate(ZooReaderWriter.java:134)
>>> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at
>>> org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:67)
>>> at com.sun.proxy.$Proxy10.mutate(Unknown Source)
>>> at
>>> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:116)
>>> ... 6 more
>>> 2014-02-27 13:32:12,327 [tableOps.Utils] INFO : table 3n
>>> (1a33c6891b997e71) unlocked for write
>>>
>>>
>>> -Russ
>>>
>>>
>>>
>>>
>>> On Thu, Feb 27, 2014 at 10:47 AM, Keith Turner <ke...@deenlo.com> wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Wed, Feb 26, 2014 at 11:10 PM, Russ Weeks <rw...@newbrightidea.com>wrote:
>>>>
>>>>> Hi, All,
>>>>>
>>>>> I can't delete any tables through the accumulo shell, I keep getting
>>>>> the message: "Internal error processing waitForTableOperation".
>>>>>
>>>>> Recently my accumulo cluster has started showing this error repeatedly
>>>>> in the master log:
>>>>> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< :
>>>>> [] 9223372036854775807 false= loc243f8e8ad6f5875>. 10.128.1.17:9997~tab ~pr-� �}
>>>>>
>>>>> And this error in the monitor log:
>>>>> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet
>>>>> !0;!0< has no prev end row
>>>>>
>>>>> And this in the GC log:
>>>>> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no
>>>>> prev end row
>>>>> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
>>>>> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>>>>>
>>>>> Maybe coincidence, but it seems like things started getting weird when
>>>>> I added a WholeRowIterator through the accumulo shell (which is nonsense, I
>>>>> know). The iterators on one of my tables (36M entries) look like this:
>>>>>
>>>>
>>>> Was that iterator added at the table level or system level? Adding at
>>>> the system level could impact the metadata table, see ACCUMULO-324.
>>>>
>>>>
>>>>>
>>>>> - Iterator WholeRowIterator, scan scope options:
>>>>> - iteratorPriority = 50
>>>>> - iteratorClassName =
>>>>> org.apache.accumulo.core.iterators.user.WholeRowIterator
>>>>> - Iterator rowdel, majc scope options:
>>>>> - iteratorPriority = 30
>>>>> - iteratorClassName =
>>>>> org.apache.accumulo.core.iterators.user.RowDeletingIterator
>>>>> - Iterator transformer, majc scope options:
>>>>> - iteratorPriority = 15
>>>>> - iteratorClassName =
>>>>> com.phemi.agile.datapolicy.DataColumnTransformer
>>>>> - maxBufferSize = 256M
>>>>> - Iterator vers, majc scope options:
>>>>> - iteratorPriority = 20
>>>>> - iteratorClassName =
>>>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>>>> - maxVersions = 1
>>>>> - Iterator vers, minc scope options:
>>>>> - iteratorPriority = 20
>>>>> - iteratorClassName =
>>>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>>>> - maxVersions = 1
>>>>> - Iterator vers, scan scope options:
>>>>> - iteratorPriority = 20
>>>>> - iteratorClassName =
>>>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>>>> - maxVersions = 1
>>>>>
>>>>> I tried deleting DataColumnTransformer, RowDeletingIterator and
>>>>> WholeRowIterator. No errors were returned from the deleteiter command in
>>>>> the shell but after running listiter, the WholeRowIterator and
>>>>> RowDeletingIterator are still there.
>>>>>
>>>>> I *am* still able to create tables, but tables that I create can't
>>>>> then be deleted.
>>>>>
>>>>> Any advice appreciated,
>>>>> -Russ
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Seeing lots of "Internal error processing waitForTableOperation"
Posted by Keith Turner <ke...@deenlo.com>.
If you run 'accumulo org.apache.accumulo.server.fate.Admin -<list option,
not sure what it is>', what does that look like?
On Thu, Feb 27, 2014 at 4:50 PM, Russ Weeks <rw...@newbrightidea.com>wrote:
> Looking at Zookeeper, it seems like some of my tables are stuck in the
> DELETING state. I can't delete these tables. I *am* able to create and
> delete new tables, or delete other tables that are not in the DELETING
> state.
>
> Is there any way to force the table out of the DELETING state?
>
> Thanks,
> -Russ
>
> [zk: localhost:2181(CONNECTED) 21] get
> /accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/state
> DELETING
> cZxid = 0x2001138c0
> ctime = Wed Feb 26 20:08:17 PST 2014
> mZxid = 0x2001138ef
> mtime = Wed Feb 26 20:08:24 PST 2014
> pZxid = 0x2001138c0
> cversion = 0
> dataVersion = 2
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 8
> numChildren = 0
> [zk: localhost:2181(CONNECTED) 22] get
> /accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/name
> test_table
> cZxid = 0x2001138bf
> ctime = Wed Feb 26 20:08:17 PST 2014
> mZxid = 0x2001138bf
> mtime = Wed Feb 26 20:08:17 PST 2014
> pZxid = 0x2001138bf
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 10
> numChildren = 0
>
>
>
>
> On Thu, Feb 27, 2014 at 1:34 PM, Russ Weeks <rw...@newbrightidea.com>wrote:
>
>> Hi, Keith,
>>
>> Thanks for your response. I think that's exactly what happened. I
>> screwed up the config command and wound up specifying a couple of iterators
>> on the !METADATA table (and indeed all tables in the system).
>>
>> I've removed those iterators using "config -d" and now I can scan the
>> !METADATA table without errors in the shell or the master log. But I'm
>> still not able to delete tables. When I run the command "deletetable
>> test_table", I see this message in the shell:
>>
>> ERROR: org.apache.accumulo.core.client.AccumuloException: Internal error
>> processing waitForTableOperation
>>
>> And this exception in the master log:
>>
>> 2014-02-27 13:32:12,311 [zookeeper.DistributedReadWriteLock] INFO : Added
>> lock entry 0 userData 1a33c6891b997e71 lockType WRITE
>> 2014-02-27 13:32:12,313 [tableOps.Utils] INFO : table 3n
>> (1a33c6891b997e71) locked for write operation: DELETE
>> 2014-02-27 13:32:12,314 [tables.TableManager] FATAL: Failed to transition
>> table to state DELETING
>> 2014-02-27 13:32:12,320 [fate.Fate] WARN : Failed to execute Repo,
>> tid=1a33c6891b997e71
>> java.lang.RuntimeException:
>> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
>> at
>> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:148)
>> at
>> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:224)
>> at
>> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:207)
>> at
>> org.apache.accumulo.server.master.tableOps.TraceRepo.call(TraceRepo.java:65)
>> at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:64)
>> at
>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34)
>> at java.lang.Thread.run(Thread.java:724)
>> Caused by:
>> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
>> at
>> org.apache.accumulo.server.master.state.tables.TableManager$1.mutate(TableManager.java:141)
>> at
>> org.apache.accumulo.fate.zookeeper.ZooReaderWriter.mutate(ZooReaderWriter.java:134)
>> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:67)
>> at com.sun.proxy.$Proxy10.mutate(Unknown Source)
>> at
>> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:116)
>> ... 6 more
>> 2014-02-27 13:32:12,327 [tableOps.Utils] INFO : table 3n
>> (1a33c6891b997e71) unlocked for write
>>
>>
>> -Russ
>>
>>
>>
>>
>> On Thu, Feb 27, 2014 at 10:47 AM, Keith Turner <ke...@deenlo.com> wrote:
>>
>>>
>>>
>>>
>>> On Wed, Feb 26, 2014 at 11:10 PM, Russ Weeks <rw...@newbrightidea.com>wrote:
>>>
>>>> Hi, All,
>>>>
>>>> I can't delete any tables through the accumulo shell, I keep getting
>>>> the message: "Internal error processing waitForTableOperation".
>>>>
>>>> Recently my accumulo cluster has started showing this error repeatedly
>>>> in the master log:
>>>> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< :
>>>> [] 9223372036854775807 false= loc243f8e8ad6f5875>. 10.128.1.17:9997~tab ~pr-� �}
>>>>
>>>> And this error in the monitor log:
>>>> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet
>>>> !0;!0< has no prev end row
>>>>
>>>> And this in the GC log:
>>>> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no
>>>> prev end row
>>>> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
>>>> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>>>>
>>>> Maybe coincidence, but it seems like things started getting weird when
>>>> I added a WholeRowIterator through the accumulo shell (which is nonsense, I
>>>> know). The iterators on one of my tables (36M entries) look like this:
>>>>
>>>
>>> Was that iterator added at the table level or system level? Adding at
>>> the system level could impact the metadata table, see ACCUMULO-324.
>>>
>>>
>>>>
>>>> - Iterator WholeRowIterator, scan scope options:
>>>> - iteratorPriority = 50
>>>> - iteratorClassName =
>>>> org.apache.accumulo.core.iterators.user.WholeRowIterator
>>>> - Iterator rowdel, majc scope options:
>>>> - iteratorPriority = 30
>>>> - iteratorClassName =
>>>> org.apache.accumulo.core.iterators.user.RowDeletingIterator
>>>> - Iterator transformer, majc scope options:
>>>> - iteratorPriority = 15
>>>> - iteratorClassName =
>>>> com.phemi.agile.datapolicy.DataColumnTransformer
>>>> - maxBufferSize = 256M
>>>> - Iterator vers, majc scope options:
>>>> - iteratorPriority = 20
>>>> - iteratorClassName =
>>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>>> - maxVersions = 1
>>>> - Iterator vers, minc scope options:
>>>> - iteratorPriority = 20
>>>> - iteratorClassName =
>>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>>> - maxVersions = 1
>>>> - Iterator vers, scan scope options:
>>>> - iteratorPriority = 20
>>>> - iteratorClassName =
>>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>>> - maxVersions = 1
>>>>
>>>> I tried deleting DataColumnTransformer, RowDeletingIterator and
>>>> WholeRowIterator. No errors were returned from the deleteiter command in
>>>> the shell but after running listiter, the WholeRowIterator and
>>>> RowDeletingIterator are still there.
>>>>
>>>> I *am* still able to create tables, but tables that I create can't then
>>>> be deleted.
>>>>
>>>> Any advice appreciated,
>>>> -Russ
>>>>
>>>>
>>>
>>
>
Re: Seeing lots of "Internal error processing waitForTableOperation"
Posted by Russ Weeks <rw...@newbrightidea.com>.
Looking at Zookeeper, it seems like some of my tables are stuck in the
DELETING state. I can't delete these tables. I *am* able to create and
delete new tables, or delete other tables that are not in the DELETING
state.
Is there any way to force the table out of the DELETING state?
Thanks,
-Russ
[zk: localhost:2181(CONNECTED) 21] get
/accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/state
DELETING
cZxid = 0x2001138c0
ctime = Wed Feb 26 20:08:17 PST 2014
mZxid = 0x2001138ef
mtime = Wed Feb 26 20:08:24 PST 2014
pZxid = 0x2001138c0
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 8
numChildren = 0
[zk: localhost:2181(CONNECTED) 22] get
/accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/name
test_table
cZxid = 0x2001138bf
ctime = Wed Feb 26 20:08:17 PST 2014
mZxid = 0x2001138bf
mtime = Wed Feb 26 20:08:17 PST 2014
pZxid = 0x2001138bf
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 10
numChildren = 0
On Thu, Feb 27, 2014 at 1:34 PM, Russ Weeks <rw...@newbrightidea.com>wrote:
> Hi, Keith,
>
> Thanks for your response. I think that's exactly what happened. I screwed
> up the config command and wound up specifying a couple of iterators on the
> !METADATA table (and indeed all tables in the system).
>
> I've removed those iterators using "config -d" and now I can scan the
> !METADATA table without errors in the shell or the master log. But I'm
> still not able to delete tables. When I run the command "deletetable
> test_table", I see this message in the shell:
>
> ERROR: org.apache.accumulo.core.client.AccumuloException: Internal error
> processing waitForTableOperation
>
> And this exception in the master log:
>
> 2014-02-27 13:32:12,311 [zookeeper.DistributedReadWriteLock] INFO : Added
> lock entry 0 userData 1a33c6891b997e71 lockType WRITE
> 2014-02-27 13:32:12,313 [tableOps.Utils] INFO : table 3n
> (1a33c6891b997e71) locked for write operation: DELETE
> 2014-02-27 13:32:12,314 [tables.TableManager] FATAL: Failed to transition
> table to state DELETING
> 2014-02-27 13:32:12,320 [fate.Fate] WARN : Failed to execute Repo,
> tid=1a33c6891b997e71
> java.lang.RuntimeException:
> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
> at
> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:148)
> at
> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:224)
> at
> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:207)
> at
> org.apache.accumulo.server.master.tableOps.TraceRepo.call(TraceRepo.java:65)
> at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:64)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at java.lang.Thread.run(Thread.java:724)
> Caused by:
> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
> at
> org.apache.accumulo.server.master.state.tables.TableManager$1.mutate(TableManager.java:141)
> at
> org.apache.accumulo.fate.zookeeper.ZooReaderWriter.mutate(ZooReaderWriter.java:134)
> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:67)
> at com.sun.proxy.$Proxy10.mutate(Unknown Source)
> at
> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:116)
> ... 6 more
> 2014-02-27 13:32:12,327 [tableOps.Utils] INFO : table 3n
> (1a33c6891b997e71) unlocked for write
>
>
> -Russ
>
>
>
>
> On Thu, Feb 27, 2014 at 10:47 AM, Keith Turner <ke...@deenlo.com> wrote:
>
>>
>>
>>
>> On Wed, Feb 26, 2014 at 11:10 PM, Russ Weeks <rw...@newbrightidea.com>wrote:
>>
>>> Hi, All,
>>>
>>> I can't delete any tables through the accumulo shell, I keep getting the
>>> message: "Internal error processing waitForTableOperation".
>>>
>>> Recently my accumulo cluster has started showing this error repeatedly
>>> in the master log:
>>> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< : []
>>> 9223372036854775807 false= loc243f8e8ad6f5875>. 10.128.1.17:9997 ~tab
>>> ~pr-� �}
>>>
>>> And this error in the monitor log:
>>> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet !0;!0<
>>> has no prev end row
>>>
>>> And this in the GC log:
>>> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no
>>> prev end row
>>> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
>>> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>>>
>>> Maybe coincidence, but it seems like things started getting weird when I
>>> added a WholeRowIterator through the accumulo shell (which is nonsense, I
>>> know). The iterators on one of my tables (36M entries) look like this:
>>>
>>
>> Was that iterator added at the table level or system level? Adding at
>> the system level could impact the metadata table, see ACCUMULO-324.
>>
>>
>>>
>>> - Iterator WholeRowIterator, scan scope options:
>>> - iteratorPriority = 50
>>> - iteratorClassName =
>>> org.apache.accumulo.core.iterators.user.WholeRowIterator
>>> - Iterator rowdel, majc scope options:
>>> - iteratorPriority = 30
>>> - iteratorClassName =
>>> org.apache.accumulo.core.iterators.user.RowDeletingIterator
>>> - Iterator transformer, majc scope options:
>>> - iteratorPriority = 15
>>> - iteratorClassName =
>>> com.phemi.agile.datapolicy.DataColumnTransformer
>>> - maxBufferSize = 256M
>>> - Iterator vers, majc scope options:
>>> - iteratorPriority = 20
>>> - iteratorClassName =
>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>> - maxVersions = 1
>>> - Iterator vers, minc scope options:
>>> - iteratorPriority = 20
>>> - iteratorClassName =
>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>> - maxVersions = 1
>>> - Iterator vers, scan scope options:
>>> - iteratorPriority = 20
>>> - iteratorClassName =
>>> org.apache.accumulo.core.iterators.user.VersioningIterator
>>> - maxVersions = 1
>>>
>>> I tried deleting DataColumnTransformer, RowDeletingIterator and
>>> WholeRowIterator. No errors were returned from the deleteiter command in
>>> the shell but after running listiter, the WholeRowIterator and
>>> RowDeletingIterator are still there.
>>>
>>> I *am* still able to create tables, but tables that I create can't then
>>> be deleted.
>>>
>>> Any advice appreciated,
>>> -Russ
>>>
>>>
>>
>
Re: Seeing lots of "Internal error processing waitForTableOperation"
Posted by Russ Weeks <rw...@newbrightidea.com>.
Hi, Keith,
Thanks for your response. I think that's exactly what happened. I screwed
up the config command and wound up specifying a couple of iterators on the
!METADATA table (and indeed all tables in the system).
I've removed those iterators using "config -d" and now I can scan the
!METADATA table without errors in the shell or the master log. But I'm
still not able to delete tables. When I run the command "deletetable
test_table", I see this message in the shell:
ERROR: org.apache.accumulo.core.client.AccumuloException: Internal error
processing waitForTableOperation
And this exception in the master log:
2014-02-27 13:32:12,311 [zookeeper.DistributedReadWriteLock] INFO : Added
lock entry 0 userData 1a33c6891b997e71 lockType WRITE
2014-02-27 13:32:12,313 [tableOps.Utils] INFO : table 3n (1a33c6891b997e71)
locked for write operation: DELETE
2014-02-27 13:32:12,314 [tables.TableManager] FATAL: Failed to transition
table to state DELETING
2014-02-27 13:32:12,320 [fate.Fate] WARN : Failed to execute Repo,
tid=1a33c6891b997e71
java.lang.RuntimeException:
org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
at
org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:148)
at
org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:224)
at
org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:207)
at
org.apache.accumulo.server.master.tableOps.TraceRepo.call(TraceRepo.java:65)
at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:64)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:724)
Caused by:
org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
at
org.apache.accumulo.server.master.state.tables.TableManager$1.mutate(TableManager.java:141)
at
org.apache.accumulo.fate.zookeeper.ZooReaderWriter.mutate(ZooReaderWriter.java:134)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:67)
at com.sun.proxy.$Proxy10.mutate(Unknown Source)
at
org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:116)
... 6 more
2014-02-27 13:32:12,327 [tableOps.Utils] INFO : table 3n (1a33c6891b997e71)
unlocked for write
-Russ
On Thu, Feb 27, 2014 at 10:47 AM, Keith Turner <ke...@deenlo.com> wrote:
>
>
>
> On Wed, Feb 26, 2014 at 11:10 PM, Russ Weeks <rw...@newbrightidea.com>wrote:
>
>> Hi, All,
>>
>> I can't delete any tables through the accumulo shell, I keep getting the
>> message: "Internal error processing waitForTableOperation".
>>
>> Recently my accumulo cluster has started showing this error repeatedly in
>> the master log:
>> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< : []
>> 9223372036854775807 false= loc243f8e8ad6f5875>. 10.128.1.17:9997 ~tab
>> ~pr-� �}
>>
>> And this error in the monitor log:
>> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet !0;!0<
>> has no prev end row
>>
>> And this in the GC log:
>> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no prev
>> end row
>> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
>> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>>
>> Maybe coincidence, but it seems like things started getting weird when I
>> added a WholeRowIterator through the accumulo shell (which is nonsense, I
>> know). The iterators on one of my tables (36M entries) look like this:
>>
>
> Was that iterator added at the table level or system level? Adding at the
> system level could impact the metadata table, see ACCUMULO-324.
>
>
>>
>> - Iterator WholeRowIterator, scan scope options:
>> - iteratorPriority = 50
>> - iteratorClassName =
>> org.apache.accumulo.core.iterators.user.WholeRowIterator
>> - Iterator rowdel, majc scope options:
>> - iteratorPriority = 30
>> - iteratorClassName =
>> org.apache.accumulo.core.iterators.user.RowDeletingIterator
>> - Iterator transformer, majc scope options:
>> - iteratorPriority = 15
>> - iteratorClassName =
>> com.phemi.agile.datapolicy.DataColumnTransformer
>> - maxBufferSize = 256M
>> - Iterator vers, majc scope options:
>> - iteratorPriority = 20
>> - iteratorClassName =
>> org.apache.accumulo.core.iterators.user.VersioningIterator
>> - maxVersions = 1
>> - Iterator vers, minc scope options:
>> - iteratorPriority = 20
>> - iteratorClassName =
>> org.apache.accumulo.core.iterators.user.VersioningIterator
>> - maxVersions = 1
>> - Iterator vers, scan scope options:
>> - iteratorPriority = 20
>> - iteratorClassName =
>> org.apache.accumulo.core.iterators.user.VersioningIterator
>> - maxVersions = 1
>>
>> I tried deleting DataColumnTransformer, RowDeletingIterator and
>> WholeRowIterator. No errors were returned from the deleteiter command in
>> the shell but after running listiter, the WholeRowIterator and
>> RowDeletingIterator are still there.
>>
>> I *am* still able to create tables, but tables that I create can't then
>> be deleted.
>>
>> Any advice appreciated,
>> -Russ
>>
>>
>
Re: Seeing lots of "Internal error processing waitForTableOperation"
Posted by Keith Turner <ke...@deenlo.com>.
On Wed, Feb 26, 2014 at 11:10 PM, Russ Weeks <rw...@newbrightidea.com>wrote:
> Hi, All,
>
> I can't delete any tables through the accumulo shell, I keep getting the
> message: "Internal error processing waitForTableOperation".
>
> Recently my accumulo cluster has started showing this error repeatedly in
> the master log:
> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< : []
> 9223372036854775807 false= loc243f8e8ad6f5875>. 10.128.1.17:9997 ~tab
> ~pr-� �}
>
> And this error in the monitor log:
> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet !0;!0<
> has no prev end row
>
> And this in the GC log:
> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no prev
> end row
> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>
> Maybe coincidence, but it seems like things started getting weird when I
> added a WholeRowIterator through the accumulo shell (which is nonsense, I
> know). The iterators on one of my tables (36M entries) look like this:
>
Was that iterator added at the table level or system level? Adding at the
system level could impact the metadata table, see ACCUMULO-324.
>
> - Iterator WholeRowIterator, scan scope options:
> - iteratorPriority = 50
> - iteratorClassName =
> org.apache.accumulo.core.iterators.user.WholeRowIterator
> - Iterator rowdel, majc scope options:
> - iteratorPriority = 30
> - iteratorClassName =
> org.apache.accumulo.core.iterators.user.RowDeletingIterator
> - Iterator transformer, majc scope options:
> - iteratorPriority = 15
> - iteratorClassName =
> com.phemi.agile.datapolicy.DataColumnTransformer
> - maxBufferSize = 256M
> - Iterator vers, majc scope options:
> - iteratorPriority = 20
> - iteratorClassName =
> org.apache.accumulo.core.iterators.user.VersioningIterator
> - maxVersions = 1
> - Iterator vers, minc scope options:
> - iteratorPriority = 20
> - iteratorClassName =
> org.apache.accumulo.core.iterators.user.VersioningIterator
> - maxVersions = 1
> - Iterator vers, scan scope options:
> - iteratorPriority = 20
> - iteratorClassName =
> org.apache.accumulo.core.iterators.user.VersioningIterator
> - maxVersions = 1
>
> I tried deleting DataColumnTransformer, RowDeletingIterator and
> WholeRowIterator. No errors were returned from the deleteiter command in
> the shell but after running listiter, the WholeRowIterator and
> RowDeletingIterator are still there.
>
> I *am* still able to create tables, but tables that I create can't then be
> deleted.
>
> Any advice appreciated,
> -Russ
>
>
Re: Seeing lots of "Internal error processing waitForTableOperation"
Posted by Russ Weeks <rw...@newbrightidea.com>.
Hi, Josh,
No, I can't scan the !METADATA table. I get about 2k of output
(here<https://gist.github.com/rweeks/055f0f19c5b959af8afc>,
if useful) and then the shell blocks. I'm seeing a lot of these exceptions
in the monitor log:
Caused by:
org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:212)
at
org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:82)
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:164)
... 6 more
-Russ
On Wed, Feb 26, 2014 at 9:33 PM, Josh Elser <jo...@gmail.com> wrote:
> Hrm, looking at the code, it appears that the !METADATA table has some
> inconsistencies. A (at least?) one row lacks of the "prev-row" column.
> Something like:
>
> some_row ~tab:~pr
>
> Sadly, I'm not positive about the cause given the information. Can you
> scan all of the !METADATA table without error?
>
> That said, I don't think that the above would prevent you from being able
> to change the iterators on a table because that's stored purely in
> ZooKeeper. Maybe it tries to read some data from !METADATA before doing the
> ZooKeeper update?
>
>
> On 2/26/14, 11:49 PM, Russ Weeks wrote:
>
>> Hi, Josh,
>>
>> Thanks for responding so quickly... sorry, I should've mentioned that
>> I'm running Accumulo 1.5.0.
>>
>> Nothing really helpful that I can see in the monitor or debug monitor
>> logs. It just interleaves the 2 errors above from the
>> state.MetaDataTableScanner and util.TabletIterator loggers.
>>
>> -Russ
>>
>>
>> On Wed, Feb 26, 2014 at 8:37 PM, Josh Elser <josh.elser@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Hi Russ,
>>
>> What version of Accumulo are you running?
>>
>> Any other exceptions in the Monitor?
>>
>> - Josh
>>
>>
>> On 2/26/14, 11:10 PM, Russ Weeks wrote:
>>
>> Hi, All,
>>
>> I can't delete any tables through the accumulo shell, I keep
>> getting the
>> message: "Internal error processing waitForTableOperation".
>>
>> Recently my accumulo cluster has started showing this error
>> repeatedly
>> in the master log:
>> [state.MetaDataTableScanner] WARN : No prev-row for key extent:
>> {z< : []
>> 9223372036854775807
>> false= loc243f8e8ad6f5875>. 10.128.1.17:9997
>> <http://10.128.1.17:9997> ~tab ~pr-� �}
>>
>>
>> And this error in the monitor log:
>> [util.TabletIterator] WARN : gc:agile01 Metadata problem :
>> tablet !0;!0<
>> has no prev end row
>>
>> And this in the GC log:
>> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has
>> no
>> prev end row
>> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 :
>> []
>> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>>
>> Maybe coincidence, but it seems like things started getting
>> weird when I
>> added a WholeRowIterator through the accumulo shell (which is
>> nonsense,
>> I know). The iterators on one of my tables (36M entries) look
>> like this:
>>
>> - Iterator WholeRowIterator, scan scope options:
>> - iteratorPriority = 50
>> - iteratorClassName =
>> org.apache.accumulo.core.__iterators.user.__WholeRowIterator
>>
>> - Iterator rowdel, majc scope options:
>> - iteratorPriority = 30
>> - iteratorClassName =
>> org.apache.accumulo.core.__iterators.user.__RowDeletingIterator
>>
>> - Iterator transformer, majc scope options:
>> - iteratorPriority = 15
>> - iteratorClassName =
>> com.phemi.agile.datapolicy.__DataColumnTransformer
>>
>> - maxBufferSize = 256M
>> - Iterator vers, majc scope options:
>> - iteratorPriority = 20
>> - iteratorClassName =
>> org.apache.accumulo.core.__iterators.user.__VersioningIterator
>>
>> - maxVersions = 1
>> - Iterator vers, minc scope options:
>> - iteratorPriority = 20
>> - iteratorClassName =
>> org.apache.accumulo.core.__iterators.user.__VersioningIterator
>>
>> - maxVersions = 1
>> - Iterator vers, scan scope options:
>> - iteratorPriority = 20
>> - iteratorClassName =
>> org.apache.accumulo.core.__iterators.user.__VersioningIterator
>>
>> - maxVersions = 1
>>
>> I tried deleting DataColumnTransformer, RowDeletingIterator and
>> WholeRowIterator. No errors were returned from the deleteiter
>> command in
>> the shell but after running listiter, the WholeRowIterator and
>> RowDeletingIterator are still there.
>>
>> I *am* still able to create tables, but tables that I create
>> can't then
>> be deleted.
>>
>> Any advice appreciated,
>> -Russ
>>
>>
>>
Re: Seeing lots of "Internal error processing waitForTableOperation"
Posted by Josh Elser <jo...@gmail.com>.
Hrm, looking at the code, it appears that the !METADATA table has some
inconsistencies. A (at least?) one row lacks of the "prev-row" column.
Something like:
some_row ~tab:~pr
Sadly, I'm not positive about the cause given the information. Can you
scan all of the !METADATA table without error?
That said, I don't think that the above would prevent you from being
able to change the iterators on a table because that's stored purely in
ZooKeeper. Maybe it tries to read some data from !METADATA before doing
the ZooKeeper update?
On 2/26/14, 11:49 PM, Russ Weeks wrote:
> Hi, Josh,
>
> Thanks for responding so quickly... sorry, I should've mentioned that
> I'm running Accumulo 1.5.0.
>
> Nothing really helpful that I can see in the monitor or debug monitor
> logs. It just interleaves the 2 errors above from the
> state.MetaDataTableScanner and util.TabletIterator loggers.
>
> -Russ
>
>
> On Wed, Feb 26, 2014 at 8:37 PM, Josh Elser <josh.elser@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi Russ,
>
> What version of Accumulo are you running?
>
> Any other exceptions in the Monitor?
>
> - Josh
>
>
> On 2/26/14, 11:10 PM, Russ Weeks wrote:
>
> Hi, All,
>
> I can't delete any tables through the accumulo shell, I keep
> getting the
> message: "Internal error processing waitForTableOperation".
>
> Recently my accumulo cluster has started showing this error
> repeatedly
> in the master log:
> [state.MetaDataTableScanner] WARN : No prev-row for key extent:
> {z< : []
> 9223372036854775807
> false= loc243f8e8ad6f5875>. 10.128.1.17:9997
> <http://10.128.1.17:9997> ~tab ~pr-� �}
>
> And this error in the monitor log:
> [util.TabletIterator] WARN : gc:agile01 Metadata problem :
> tablet !0;!0<
> has no prev end row
>
> And this in the GC log:
> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no
> prev end row
> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>
> Maybe coincidence, but it seems like things started getting
> weird when I
> added a WholeRowIterator through the accumulo shell (which is
> nonsense,
> I know). The iterators on one of my tables (36M entries) look
> like this:
>
> - Iterator WholeRowIterator, scan scope options:
> - iteratorPriority = 50
> - iteratorClassName =
> org.apache.accumulo.core.__iterators.user.__WholeRowIterator
> - Iterator rowdel, majc scope options:
> - iteratorPriority = 30
> - iteratorClassName =
> org.apache.accumulo.core.__iterators.user.__RowDeletingIterator
> - Iterator transformer, majc scope options:
> - iteratorPriority = 15
> - iteratorClassName =
> com.phemi.agile.datapolicy.__DataColumnTransformer
> - maxBufferSize = 256M
> - Iterator vers, majc scope options:
> - iteratorPriority = 20
> - iteratorClassName =
> org.apache.accumulo.core.__iterators.user.__VersioningIterator
> - maxVersions = 1
> - Iterator vers, minc scope options:
> - iteratorPriority = 20
> - iteratorClassName =
> org.apache.accumulo.core.__iterators.user.__VersioningIterator
> - maxVersions = 1
> - Iterator vers, scan scope options:
> - iteratorPriority = 20
> - iteratorClassName =
> org.apache.accumulo.core.__iterators.user.__VersioningIterator
> - maxVersions = 1
>
> I tried deleting DataColumnTransformer, RowDeletingIterator and
> WholeRowIterator. No errors were returned from the deleteiter
> command in
> the shell but after running listiter, the WholeRowIterator and
> RowDeletingIterator are still there.
>
> I *am* still able to create tables, but tables that I create
> can't then
> be deleted.
>
> Any advice appreciated,
> -Russ
>
>
Re: Seeing lots of "Internal error processing waitForTableOperation"
Posted by Russ Weeks <rw...@newbrightidea.com>.
Hi, Josh,
Thanks for responding so quickly... sorry, I should've mentioned that I'm
running Accumulo 1.5.0.
Nothing really helpful that I can see in the monitor or debug monitor logs.
It just interleaves the 2 errors above from the state.MetaDataTableScanner
and util.TabletIterator loggers.
-Russ
On Wed, Feb 26, 2014 at 8:37 PM, Josh Elser <jo...@gmail.com> wrote:
> Hi Russ,
>
> What version of Accumulo are you running?
>
> Any other exceptions in the Monitor?
>
> - Josh
>
>
> On 2/26/14, 11:10 PM, Russ Weeks wrote:
>
>> Hi, All,
>>
>> I can't delete any tables through the accumulo shell, I keep getting the
>> message: "Internal error processing waitForTableOperation".
>>
>> Recently my accumulo cluster has started showing this error repeatedly
>> in the master log:
>> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< : []
>> 9223372036854775807
>> false= loc243f8e8ad6f5875>. 10.128.1.17:9997 ~tab ~pr-� �}
>>
>> And this error in the monitor log:
>> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet !0;!0<
>> has no prev end row
>>
>> And this in the GC log:
>> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no
>> prev end row
>> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
>> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>>
>> Maybe coincidence, but it seems like things started getting weird when I
>> added a WholeRowIterator through the accumulo shell (which is nonsense,
>> I know). The iterators on one of my tables (36M entries) look like this:
>>
>> - Iterator WholeRowIterator, scan scope options:
>> - iteratorPriority = 50
>> - iteratorClassName =
>> org.apache.accumulo.core.iterators.user.WholeRowIterator
>> - Iterator rowdel, majc scope options:
>> - iteratorPriority = 30
>> - iteratorClassName =
>> org.apache.accumulo.core.iterators.user.RowDeletingIterator
>> - Iterator transformer, majc scope options:
>> - iteratorPriority = 15
>> - iteratorClassName =
>> com.phemi.agile.datapolicy.DataColumnTransformer
>> - maxBufferSize = 256M
>> - Iterator vers, majc scope options:
>> - iteratorPriority = 20
>> - iteratorClassName =
>> org.apache.accumulo.core.iterators.user.VersioningIterator
>> - maxVersions = 1
>> - Iterator vers, minc scope options:
>> - iteratorPriority = 20
>> - iteratorClassName =
>> org.apache.accumulo.core.iterators.user.VersioningIterator
>> - maxVersions = 1
>> - Iterator vers, scan scope options:
>> - iteratorPriority = 20
>> - iteratorClassName =
>> org.apache.accumulo.core.iterators.user.VersioningIterator
>> - maxVersions = 1
>>
>> I tried deleting DataColumnTransformer, RowDeletingIterator and
>> WholeRowIterator. No errors were returned from the deleteiter command in
>> the shell but after running listiter, the WholeRowIterator and
>> RowDeletingIterator are still there.
>>
>> I *am* still able to create tables, but tables that I create can't then
>> be deleted.
>>
>> Any advice appreciated,
>> -Russ
>>
>>
Re: Seeing lots of "Internal error processing waitForTableOperation"
Posted by Josh Elser <jo...@gmail.com>.
Hi Russ,
What version of Accumulo are you running?
Any other exceptions in the Monitor?
- Josh
On 2/26/14, 11:10 PM, Russ Weeks wrote:
> Hi, All,
>
> I can't delete any tables through the accumulo shell, I keep getting the
> message: "Internal error processing waitForTableOperation".
>
> Recently my accumulo cluster has started showing this error repeatedly
> in the master log:
> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< : []
> 9223372036854775807
> false=loc243f8e8ad6f5875>.10.128.1.17:9997~tab~pr-��}
>
> And this error in the monitor log:
> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet !0;!0<
> has no prev end row
>
> And this in the GC log:
> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no
> prev end row
> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
> 9223372036854775807 false,~ : [] 9223372036854775807 false)
>
> Maybe coincidence, but it seems like things started getting weird when I
> added a WholeRowIterator through the accumulo shell (which is nonsense,
> I know). The iterators on one of my tables (36M entries) look like this:
>
> - Iterator WholeRowIterator, scan scope options:
> - iteratorPriority = 50
> - iteratorClassName =
> org.apache.accumulo.core.iterators.user.WholeRowIterator
> - Iterator rowdel, majc scope options:
> - iteratorPriority = 30
> - iteratorClassName =
> org.apache.accumulo.core.iterators.user.RowDeletingIterator
> - Iterator transformer, majc scope options:
> - iteratorPriority = 15
> - iteratorClassName =
> com.phemi.agile.datapolicy.DataColumnTransformer
> - maxBufferSize = 256M
> - Iterator vers, majc scope options:
> - iteratorPriority = 20
> - iteratorClassName =
> org.apache.accumulo.core.iterators.user.VersioningIterator
> - maxVersions = 1
> - Iterator vers, minc scope options:
> - iteratorPriority = 20
> - iteratorClassName =
> org.apache.accumulo.core.iterators.user.VersioningIterator
> - maxVersions = 1
> - Iterator vers, scan scope options:
> - iteratorPriority = 20
> - iteratorClassName =
> org.apache.accumulo.core.iterators.user.VersioningIterator
> - maxVersions = 1
>
> I tried deleting DataColumnTransformer, RowDeletingIterator and
> WholeRowIterator. No errors were returned from the deleteiter command in
> the shell but after running listiter, the WholeRowIterator and
> RowDeletingIterator are still there.
>
> I *am* still able to create tables, but tables that I create can't then
> be deleted.
>
> Any advice appreciated,
> -Russ
>