[jira] [Commented] (ACCUMULO-2466) Bulk randomwalk fails with bad key

["Bill Havanki (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ACCUMULO-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933491#comment-13933491 ] 

Bill Havanki commented on ACCUMULO-2466:
----------------------------------------

There are 121 markers for row r00000, each with value 1. Marker 00018 is missing. That is one of the BulkMinusOne steps that failed. The other two correspond to markers 00019 and 00022, which are somehow present in the table.

> Bulk randomwalk fails with bad key
> ----------------------------------
>
>                 Key: ACCUMULO-2466
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2466
>             Project: Accumulo
>          Issue Type: Bug
>          Components: master, test
>    Affects Versions: 1.4.4
>            Reporter: Bill Havanki
>              Labels: import, randomwalk, test
>
> Running bulk randomwalk against 1.4.5-SNAPSHOT, got this in verification:
> {noformat}
> Caused by: java.lang.Exception: Bad key at r00000 cf:000 [] 1394658887772 false 1
>         at org.apache.accumulo.server.test.randomwalk.bulk.Verify.visit(Verify.java:65)
> {noformat}
> Possible reasons:
> * ACCUMULO-2110, not backported to 1.4 or 1.5
> * master agitation
> I see in the logs three internal errors from imports that failed due to the masters being restarted. The failure timing is around 5 seconds after the masters restart. Example:
> {noformat}
> 12 14:10:17,580 [bulk.BulkMinusOne] ERROR: org.apache.accumulo.core.client.AccumuloException: Intern
> al error processing waitForTableOperation
> org.apache.accumulo.core.client.AccumuloException: Internal error processing waitForTableOperation
>         at org.apache.accumulo.core.client.admin.TableOperationsImpl.doTableOperation(TableOperation
> sImpl.java:290)
>         at org.apache.accumulo.core.client.admin.TableOperationsImpl.doTableOperation(TableOperation
> sImpl.java:258)
>         at org.apache.accumulo.core.client.admin.TableOperationsImpl.importDirectory(TableOperations
> Impl.java:947)
>         at org.apache.accumulo.server.test.randomwalk.bulk.BulkPlusOne.bulkLoadLots(BulkPlusOne.java
> :99)
>         at org.apache.accumulo.server.test.randomwalk.bulk.BulkMinusOne.runLater(BulkMinusOne.java:2
> 9)
> ...
> Caused by: org.apache.thrift.TApplicationException: Internal error processing waitForTableOperation
> {noformat}
> Two BulkMinusOne and one BulkPlusOne failed, which may be why the offending row was at value 1.
> The {{TableOperationsImpl.waitForTableOperation}} method does not catch {{TApplicationException}}, so the imports fail.
> I see lots of previous work on this sort of error in ACCUMULO-334 and ACCUMULO-2110. If anyone has troubleshooting tips I'd be happy to hear them.



--
This message was sent by Atlassian JIRA
(v6.2#6252)