[jira] [Created] (HBASE-21139) Concurrent invocations of MetricsTableAggregateSourceImpl.getOrCreateTableSource may return unregistered MetricsTableSource

["Ted Yu (JIRA)" <ji...@apache.org>]

Ted Yu created HBASE-21139:
------------------------------

             Summary: Concurrent invocations of MetricsTableAggregateSourceImpl.getOrCreateTableSource may return unregistered MetricsTableSource
                 Key: HBASE-21139
                 URL: https://issues.apache.org/jira/browse/HBASE-21139
             Project: HBase
          Issue Type: Bug
            Reporter: Ted Yu


From test output of TestRestoreFlushSnapshotFromClient :
{code}
2018-09-01 21:09:38,174 WARN  [member: 'hw13463.attlocal.net,49623,1535861370108' subprocedure-pool6-thread-1] snapshot.                                                          RegionServerSnapshotManager$SnapshotSubprocedurePool(348): Got Exception in SnapshotSubprocedurePool
java.util.concurrent.ExecutionException: java.lang.NullPointerException
  at java.util.concurrent.FutureTask.report(FutureTask.java:122)
  at java.util.concurrent.FutureTask.get(FutureTask.java:192)
  at org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:324)
  at org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:173)
  at org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:193)
  at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:189)
  at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:53)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
  at org.apache.hadoop.hbase.regionserver.MetricsTableSourceImpl.updateFlushTime(MetricsTableSourceImpl.java:375)
  at org.apache.hadoop.hbase.regionserver.MetricsTable.updateFlushTime(MetricsTable.java:56)
  at org.apache.hadoop.hbase.regionserver.MetricsRegionServer.updateFlush(MetricsRegionServer.java:210)
  at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2826)
  at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2444)
  at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2416)
  at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2306)
  at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2209)
  at org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask.call(FlushSnapshotSubprocedure.java:115)
  at org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask.call(FlushSnapshotSubprocedure.java:77)
{code}
In MetricsTableAggregateSourceImpl.getOrCreateTableSource :
{code}
    MetricsTableSource prev = tableSources.putIfAbsent(table, source);

    if (prev != null) {
      return prev;
    } else {
      // register the new metrics now
      register(source);
{code}
Suppose threads t1 and t2 execute the above code concurrently.
t1 calls putIfAbsent first and proceeds to running {{register(source)}}.
Context switches, t2 gets to putIfAbsent and retrieves the instance stored by t1 which is not registered yet.
We would end up with what the stack trace showed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)