[jira] [Commented] (ACCUMULO-2213) tracer reports: IllegalStateException: Closed

["Sean Busbey (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ACCUMULO-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13874853#comment-13874853 ] 

Sean Busbey commented on ACCUMULO-2213:
---------------------------------------

I can make this happen reliably on 1.4.5 while using gremlins to disrupt the node the tracer is on.

I also get exceptions in the main loop, which blows up the log for the service:

{noformat}
2014-01-15 23:41:10,533 [trace.TraceServer] ERROR: Unable to write mutation to table: org.apache.accumulo.core.data.Mutation@0
java.lang.IllegalStateException: Closed
        at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addMutation(TabletServerBatchWriter.java:192)
        at org.apache.accumulo.core.client.impl.BatchWriterImpl.addMutation(BatchWriterImpl.java:40)
        at org.apache.accumulo.server.trace.TraceServer$Receiver.span(TraceServer.java:136)
        at org.apache.accumulo.cloudtrace.thrift.SpanReceiver$Processor$span.process(SpanReceiver.java:205)
        at org.apache.accumulo.cloudtrace.thrift.SpanReceiver$Processor.process(SpanReceiver.java:185)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2014-01-15 23:41:10,533 [trace.TraceServer] ERROR: Unable to write mutation to table: org.apache.accumulo.core.data.Mutation@0
java.lang.IllegalStateException: Closed
        at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addMutation(TabletServerBatchWriter.java:192)
        at org.apache.accumulo.core.client.impl.BatchWriterImpl.addMutation(BatchWriterImpl.java:40)
        at org.apache.accumulo.server.trace.TraceServer$Receiver.span(TraceServer.java:137)
        at org.apache.accumulo.cloudtrace.thrift.SpanReceiver$Processor$span.process(SpanReceiver.java:205)
        at org.apache.accumulo.cloudtrace.thrift.SpanReceiver$Processor.process(SpanReceiver.java:185)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
{noformat}

I think there are two issues here: that the writer gets closed and that our error handling to reset doesn't properly account for enough error conditions (right now it's just Mutations Rejected).

I've been talking in IRC as I try to figure out how the tracer's writer could get closed. Can we coordinate there?

> tracer reports: IllegalStateException: Closed
> ---------------------------------------------
>
>                 Key: ACCUMULO-2213
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2213
>             Project: Accumulo
>          Issue Type: Bug
>          Components: trace
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>            Priority: Minor
>             Fix For: 1.4.5, 1.5.1, 1.6.0
>
>
> During a 24 hour continuous ingest test with agitation, the following was reported 42k times:
> {noformat}
> Timer task failed org.apache.accumulo.tracer.TraceServer$1 Closed
> 	java.lang.IllegalStateException: Closed
> 		at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.flush(TabletServerBatchWriter.java:302)
> 		at org.apache.accumulo.core.client.impl.BatchWriterImpl.flush(BatchWriterImpl.java:59)
> 		at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:225)
> 		at org.apache.accumulo.tracer.TraceServer.access$400(TraceServer.java:75)
> 		at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:217)
> 		at org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
> 		at java.util.TimerThread.mainLoop(Timer.java:512)
> 		at java.util.TimerThread.run(Timer.java:462)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)