You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by "Keith Turner (Created) (JIRA)" <ji...@apache.org> on 2012/02/08 20:54:59 UTC
[jira] [Created] (ACCUMULO-383) Shard random walk test failed w/
TableDeleted exception
Shard random walk test failed w/ TableDeleted exception
-------------------------------------------------------
Key: ACCUMULO-383
URL: https://issues.apache.org/jira/browse/ACCUMULO-383
Project: Accumulo
Issue Type: Bug
Reporter: Keith Turner
While running the random walk test on a 10 node cluster, the shard test failed with the following errors.
{noformat}
07 23:43:38,483 [shard.CloneIndex] DEBUG: Cloned ST_index_26192_1328649015908_tmp from ST_index_26192_1328649015908 flush: 5825ms clone: 96ms
07 23:43:38,503 [impl.ThriftScanner] DEBUG: Failed to locate tablet for table : rr row :
07 23:49:52,795 [impl.ThriftScanner] DEBUG: Scan failed, not serving tablet (ln<<,xxx.xxx.xxx.10:9997)
07 23:51:11,539 [shard.VerifyIndex] DEBUG: Verified 65438656 index entries
07 23:51:12,052 [shard.Search] DEBUG: Looking up terms [67x, 1o3g, dl4, e4m] expect to find 2682000000000000
07 23:51:12,110 [impl.TabletServerBatchReaderIterator] DEBUG: Server : xxx.xxx.xxx.8:9997 msg : null
ThriftSecurityException(user:root, code:PERMISSION_DENIED)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startMultiScan_result.read(TabletClientService.java:7586)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startMultiScan(TabletClientService.java:306)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startMultiScan(TabletClientService.java:274)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:83)
at $Proxy2.startMultiScan(Unknown Source)
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:539)
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:335)
at cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
07 23:51:12,113 [impl.TabletServerBatchReaderIterator] DEBUG: Error PERMISSION_DENIED - User does not have permission to perform this action
org.apache.accumulo.core.client.AccumuloSecurityException: Error PERMISSION_DENIED - User does not have permission to perform this action
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:576)
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:335)
at cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: ThriftSecurityException(user:root, code:PERMISSION_DENIED)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startMultiScan_result.read(TabletClientService.java:7586)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startMultiScan(TabletClientService.java:306)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startMultiScan(TabletClientService.java:274)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:83)
at $Proxy2.startMultiScan(Unknown Source)
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:539)
... 5 more
07 23:51:12,114 [randomwalk.Module] DEBUG: Properties for node: shard.Search
07 23:51:12,114 [randomwalk.Module] DEBUG: teardown:
07 23:51:12,114 [randomwalk.Module] DEBUG: maxSec:
07 23:51:12,114 [randomwalk.Module] DEBUG: maxHops:
07 23:51:12,114 [randomwalk.Module] DEBUG: Properties for node: Shard.xml
07 23:51:12,114 [randomwalk.Module] DEBUG: teardown:
07 23:51:12,114 [randomwalk.Module] DEBUG: maxSec:
07 23:51:12,114 [randomwalk.Module] DEBUG: maxHops:
07 23:51:12,114 [randomwalk.Framework] ERROR: Error during random walk
java.lang.Exception: Error running node Shard.xml
at org.apache.accumulo.server.test.randomwalk.Module.visit(Module.java:259)
at org.apache.accumulo.server.test.randomwalk.Framework.run(Framework.java:61)
at org.apache.accumulo.server.test.randomwalk.Framework.main(Framework.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.accumulo.start.Main$1.run(Main.java:89)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.Exception: Error running node shard.Search
at org.apache.accumulo.server.test.randomwalk.Module.visit(Module.java:259)
at org.apache.accumulo.server.test.randomwalk.Module.visit(Module.java:251)
... 8 more
Caused by: org.apache.accumulo.core.client.TableDeletedException
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:357)
at cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
... 1 more
{noformat}
After seeing this looked and saw that the table existed. So for some reason a tablet server and client process did not see metadata related to the table. This happened after a delete and rename. One possibility is that the client is using the old table id from before the delete and rename. However, not sure how this could have happened. If the log messages printed the table id this would be easy to verify. Another possibility is that its some sort of zookeeper hiccup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira