[jira] [Comment Edited] (HBASE-21991) Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements

["Sakthi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783982#comment-16783982 ] 

Sakthi edited comment on HBASE-21991 at 3/5/19 2:18 AM:
--------------------------------------------------------

Regarding the race condition:
 * With multiple rpc handler threads trying to access the map of <meter names,meters>, a thread might try to access a metric that was removed by some other thread from the map as shown below:

{code:java}
   private void markMeterIfPresent(String requestMeter) {
      ...
      if (requestsMap.containsKey(requestMeter) && requestsMap.get(requestMeter).isPresent()) {    
      Meter metric = (Meter) requestsMap.get(requestMeter).get();// <-----------Thread-1 trying to access the 
      // -----------metric here that was removed by Thread-2 below----------------------------
        ...
      }
    }

    private void registerLossyCountingMeterIfNotPresent(...) {
      ...
      Set<String> metersToBeRemoved = lossyCounting.addByOne(requestMeter);
      if(!requestsMap.containsKey(requestMeter) && metersToBeRemoved.contains(requestMeter)){
        for(String meter: metersToBeRemoved) {
          requestsMap.remove(meter); //------------------> Thread-2 removing the meter here
          ...
        }
        ...
      }
      ...
    }
{code}
 

Verified with a unit test. Frequently got errors of the below kind:
{code:none}
regionserver.HRegionServer: ***** ABORTING region server x.y.z.24,16020,1548747043814: The coprocessor org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw java.lang.NullPointerException *****
{code}


was (Author: jatsakthi):
Regarding the race condition:
 * With multiple rcp handler threads trying to access the map of <meter names,meters>, a thread might try to access a metric that was removed by some other thread from the map as shown below:

{code:java}
   private void markMeterIfPresent(String requestMeter) {
      ...
      if (requestsMap.containsKey(requestMeter) && requestsMap.get(requestMeter).isPresent()) {    
      Meter metric = (Meter) requestsMap.get(requestMeter).get();// <-----------Thread-1 trying to access the 
      // -----------metric here that was removed by Thread-2 below----------------------------
        ...
      }
    }

    private void registerLossyCountingMeterIfNotPresent(...) {
      ...
      Set<String> metersToBeRemoved = lossyCounting.addByOne(requestMeter);
      if(!requestsMap.containsKey(requestMeter) && metersToBeRemoved.contains(requestMeter)){
        for(String meter: metersToBeRemoved) {
          requestsMap.remove(meter); //------------------> Thread-2 removing the meter here
          ...
        }
        ...
      }
      ...
    }
{code}
 

Verified with a unit test. Frequently got errors of the below kind:
{code:none}
regionserver.HRegionServer: ***** ABORTING region server x.y.z.24,16020,1548747043814: The coprocessor org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw java.lang.NullPointerException *****
{code}





> Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements
> --------------------------------------------------------------------------------
>
>                 Key: HBASE-21991
>                 URL: https://issues.apache.org/jira/browse/HBASE-21991
>             Project: HBase
>          Issue Type: Bug
>          Components: Coprocessors, metrics
>            Reporter: Sakthi
>            Assignee: Sakthi
>            Priority: Major
>
> Here is a list of the issues related to the MetaMetrics implementation:
> +*Bugs*+:
>  # [_Lossy counting for top-k_] *Faulty remove logic of non-eligible meters*: Under certain conditions, we might end up storing/exposing all the meters rather than top-k-ish
>  # MetaMetrics can throw NPE resulting in aborting of the RS because of a *Race Condition*.
> +*Improvements*+:
>  # With high number of regions in the cluster, exposure of metrics for each region blows up the JMX from ~140 Kbs to 100+ Mbs depending on the number of regions. It's better to use *lossy counting to maintain top-k for region metrics* as well.
>  # As the lossy meters do not represent actual counts, I think, it'll be better to *rename the meters to include "lossy" in the name*. It would be more informative while monitoring the metrics and there would be less confusion regarding actual counts to lossy counts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)