You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Qiang Tian (JIRA)" <ji...@apache.org> on 2014/11/25 03:59:12 UTC

[jira] [Commented] (HBASE-11902) RegionServer was blocked while aborting

    [ https://issues.apache.org/jira/browse/HBASE-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223976#comment-14223976 ] 

Qiang Tian commented on HBASE-11902:
------------------------------------

from the stacktrace, DrainBarrier.stopAndDrainOps is waiting, so DrainBarrier#endOp does not notify it.

looking at class DrainBarrier, it is expected that beginOp and endOp are called in pair. the initial value of {{valueAndFlags}} is 2, incremented by 2 in beginOp; decremented by 2 in endOp.

in stopAndDrainOps, if getValue(oldValAndFlags) == 1, means oldValAndFlags=2, all ops are completed in pair, otherwise, it needs to wait the last endOp to notify it:

{code}
    if (getValue(oldValAndFlags) == 1) return; // There were no operations outstanding.
    synchronized (this) { this.wait(); }
{code}

so the problem could be the beginOp/endOp is not called in pair, the hole looks to be here:

HRegion#internalFlushcache
{code}
    // sync unflushed WAL changes when deferred log sync is enabled
    // see HBASE-8208 for details
    if (wal != null && !shouldSyncLog()) {
      wal.sync();
    }
{code} 

at that point, wal.startCacheFlush->closeBarrier.beginOp is called, but completeCacheFlush->closeBarrier.endOp() is not protected by a try block..so if WAL/HDFS layer throws exception, the endOp will not be called.

related info in log:

  
{quote}    
2014-09-03 13:38:03,789 ERROR org.apache.hadoop.hbase.regionserver.wal.FSHLog: Error while AsyncWriter write, request close of hlog<
java.io.IOException: All datanodes 10.246.2.103:50010 are bad. Aborting...
  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1127)
  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:924)
  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2014-09-03 13:38:03,789 FATAL org.apache.hadoop.hbase.regionserver.wal.FSHLog: Error while AsyncSyncer sync, request close of hlog<
java.io.IOException: All datanodes 10.246.2.103:50010 are bad. Aborting...
  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1127)
  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:924)
  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2014-09-03 13:38:03,799 ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flush failed for region page_content_queue,00166,1408946731655.8671b8a0f82565f88eb2ab8a5b53e84c.  //<==========MemStoreFlusher#flushRegion
java.io.IOException: All datanodes 10.246.2.103:50010 are bad. Aborting...
{quote}


the exception thrown to caller should be here:
{code}
FSHLog#syncer:

    if (txid <= this.failedTxid.get()) {
        assert asyncIOE != null :
          "current txid is among(under) failed txids, but asyncIOE is null!";
        throw asyncIOE;
    }

{code}

the master branch can catch the hdfs exception, but it just ignore it, which looks incorrect:
{code}
      if (wal != null) {
        try {
          wal.sync(); // ensure that flush marker is sync'ed
        } catch (IOException ioe) {
          LOG.warn("Unexpected exception while wal.sync(), ignoring. Exception: "
              + StringUtils.stringifyException(ioe));
        }
      }
{code}

Personally the exeception should not be ignored since it is severe hdfs error.


> RegionServer was blocked while aborting
> ---------------------------------------
>
>                 Key: HBASE-11902
>                 URL: https://issues.apache.org/jira/browse/HBASE-11902
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver, wal
>    Affects Versions: 0.98.4
>         Environment: hbase-0.98.4, hadoop-2.3.0-cdh5.1, jdk1.7
>            Reporter: Victor Xu
>         Attachments: hbase-hadoop-regionserver-hadoop461.cm6.log, jstack_hadoop461.cm6.log
>
>
> Generally, regionserver automatically aborts when isHealth() returns false. But it sometimes got blocked while aborting. I saved the jstack and logs, and found out that it was caused by datanodes failures. The "regionserver60020" thread was blocked while closing WAL. 
> This issue doesn't happen so frequently, but if it happens, it always leads to huge amount of requests failure. The only way to do is KILL -9.
> I think it's a bug, but I haven't found a decent solution. Does anyone have the same problem?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)