[jira] [Created] (ACCUMULO-893) splitting and hold time are interacting badly

["Eric Newton (JIRA)" <ji...@apache.org>]

Eric Newton created ACCUMULO-893:
------------------------------------

             Summary: splitting and hold time are interacting badly
                 Key: ACCUMULO-893
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-893
             Project: Accumulo
          Issue Type: Bug
          Components: tserver
    Affects Versions: 1.4.2
            Reporter: Eric Newton
            Assignee: Keith Turner
             Fix For: 1.5.0


Testing on a medium sized cluster using the continuous ingest test.

I pre-split the table and started ingest.  I got tired of waiting for the tablets to split again so I lowered the split threshold to 100M.

The tablet servers started minor compaction in preparation for the split; memory became held.. and then nothing would happen until this error came up:

{noformat}
2012-12-06 12:57:02,025 [thrift.TabletClientService$Processor] ERROR: tserver:r01n16 Internal error processing closeUpdate
org.apache.accumulo.server.tabletserver.HoldTimeoutException: Commits are held
        at org.apache.accumulo.server.tabletserver.TabletServerResourceManager.waitUntilCommitsAreEnabled(TabletServerResourceManager.java:386)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.flush(TabletServer.java:1469)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.closeUpdate(TabletServer.java:1610)
        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
        at $Proxy2.closeUpdate(Unknown Source)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$closeUpdate.process(TabletClientService.java:2338)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:662)
{noformat}

The tservers appeared to make no progress on splitting until I stopped the ingesters.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira