You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2016/04/26 18:21:12 UTC
[jira] [Commented] (HBASE-15711) Add client side property to allow
logging details for batch errors
[ https://issues.apache.org/jira/browse/HBASE-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258363#comment-15258363 ]
stack commented on HBASE-15711:
-------------------------------
+1
Anything to help w/ debugging. Needs some doc though else we'll just forget it exists.
PIty we could not dynamically set this config? Can we not? Via the UI as we set log levels now? We could set a few UI configs... like this one? For another issue.
> Add client side property to allow logging details for batch errors
> ------------------------------------------------------------------
>
> Key: HBASE-15711
> URL: https://issues.apache.org/jira/browse/HBASE-15711
> Project: HBase
> Issue Type: Bug
> Reporter: Yu Li
> Assignee: Yu Li
> Attachments: HBASE-15711.patch
>
>
> Recently we observed below error message on client side when put process blocked:
> {noformat}
> 2016-04-26 10:27:11,707 ERROR [Sink: Unnamed (1/1)] com.alibaba.search.blink.streaming.connector.hbase.HBaseOutputFormat: Doing mutation failed
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1 time,
> at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:208)
> at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1694)
> at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
> at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
> {noformat}
> And checking RS logs we found nothing noticable. After adding some logging to show the detailed exception, it turns out something went wrong in one of our coprocessors:
> {noformat}
> 2016-04-26 12:03:13,776 ERROR [Sink: Unnamed (1/1)] org.apache.hadoop.hbase.client.AsyncProcess: Exception occurred! Exception details: [java.io.IOException: java.io.IOException: notify meta has not load success.
> at com.taobao.kart.coprocessor.server.CoprocessorNotifyMeta.checkMetaLoadSuccess(CoprocessorNotifyMeta.java:38)
> at com.taobao.kart.coprocessor.server.NotifyQualifySetter.updatePut(NotifyQualifySetter.java:47)
> at com.taobao.kart.coprocessor.server.NotifyQualifySetter.updatePut(NotifyQualifySetter.java:39)
> at com.taobao.kart.coprocessor.server.KartCoprocessor.prePut(KartCoprocessor.java:176)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:902)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:898)
> at org.apache.hadoop.hbase.regionserver.HRegion.doPreMutationHook(HRegion.java:2890)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2865)
> {noformat}
> So in this JIRA we propose to add a property to allow logging detailed exception stacktrace rather than statistics for batch errors, and I believe this would be useful for debugging in some cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)