You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "Josh Elser (JIRA)" <ji...@apache.org> on 2014/11/21 05:04:34 UTC
[jira] [Updated] (ACCUMULO-3351) Tracer can't write traces after
offline and online of trace table
[ https://issues.apache.org/jira/browse/ACCUMULO-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Elser updated ACCUMULO-3351:
---------------------------------
Fix Version/s: 1.7.0
> Tracer can't write traces after offline and online of trace table
> -----------------------------------------------------------------
>
> Key: ACCUMULO-3351
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3351
> Project: Accumulo
> Issue Type: Bug
> Components: trace
> Affects Versions: 1.6.1
> Reporter: Josh Elser
> Assignee: Josh Elser
> Fix For: 1.6.2, 1.7.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> While running tests for ACCUMULO-3167, I updated one of the tests to offline the trace table to reduce the possibility that any active logs for the trace table would exist in the metadata table.
> A later test went to validate that traces were found for some conditional update sessions and hung indefinitely.
> Inspecting the tracer log, the batchwriter had two exceptions due to the trace table being offline (as expected), but never recovered when the trace table came back online.
> {noformat}
> 2014-11-20 13:08:28,717 [impl.TabletServerBatchWriter] DEBUG: Table trace (in) is offline
> org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is offline
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:662)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:694)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:233)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addFailedMutations(TabletServerBatchWriter.java:551)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.access$700(TabletServerBatchWriter.java:101)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$FailedMutations.run(TabletServerBatchWriter.java:603)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> 2014-11-20 13:08:28,720 [tracer.TraceServer] WARN : Problem flushing traces, resetting writer. Set log level to DEBUG to see stacktrace. cause: org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations : 0 security codes: {} # server errors 0 # exceptions 1
> 2014-11-20 13:08:28,720 [tracer.TraceServer] DEBUG: flushing traces failed due to exception
> org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations : 0 security codes: {} # server errors 0 # exceptions 1
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:537)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.flush(TabletServerBatchWriter.java:331)
> at org.apache.accumulo.core.client.impl.BatchWriterImpl.flush(BatchWriterImpl.java:59)
> at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:245)
> at org.apache.accumulo.tracer.TraceServer.access$300(TraceServer.java:78)
> at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:235)
> at org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Caused by: org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is offline
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:662)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:694)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:233)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addFailedMutations(TabletServerBatchWriter.java:551)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.access$700(TabletServerBatchWriter.java:101)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$FailedMutations.run(TabletServerBatchWriter.java:603)
> ... 2 more
> 2014-11-20 13:08:28,722 [tracer.TraceServer] WARN : Unable to create a batch writer, will retry. Set log level to DEBUG to see stacktrace. cause: org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is offline
> 2014-11-20 13:08:28,722 [tracer.TraceServer] DEBUG: batch writer creation failed with exception.
> org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is offline
> at org.apache.accumulo.core.client.impl.ConnectorImpl.getTableId(ConnectorImpl.java:86)
> at org.apache.accumulo.core.client.impl.ConnectorImpl.createBatchWriter(ConnectorImpl.java:128)
> at org.apache.accumulo.tracer.TraceServer.resetWriter(TraceServer.java:262)
> at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:250)
> at org.apache.accumulo.tracer.TraceServer.access$300(TraceServer.java:78)
> at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:235)
> at org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> 2014-11-20 13:08:28,723 [tracer.TraceServer] WARN : Problem closing batch writer. Set log level to DEBUG to see stacktrace. cause: org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations : 0 security codes: {} # server errors 0 # exceptions 1
> 2014-11-20 13:08:28,723 [tracer.TraceServer] DEBUG: batch writer close failed with exception
> org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations : 0 security codes: {} # server errors 0 # exceptions 1
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:537)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.close(TabletServerBatchWriter.java:354)
> at org.apache.accumulo.core.client.impl.BatchWriterImpl.close(BatchWriterImpl.java:54)
> at org.apache.accumulo.tracer.TraceServer.resetWriter(TraceServer.java:271)
> at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:250)
> at org.apache.accumulo.tracer.TraceServer.access$300(TraceServer.java:78)
> at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:235)
> at org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Caused by: org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is offline
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:662)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:694)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:233)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addFailedMutations(TabletServerBatchWriter.java:551)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.access$700(TabletServerBatchWriter.java:101)
> at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$FailedMutations.run(TabletServerBatchWriter.java:603)
> ... 2 more
> 2014-11-20 13:10:12,929 [tracer.TraceServer] WARN : writer is not ready; discarding span.
> 2014-11-20 13:10:12,930 [tracer.TraceServer] WARN : writer is not ready; discarding span.
> {noformat}
> "writer is not ready; discarding span." repeats indefinitely.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)