[jira] [Updated] (TRAFODION-2606) Regionserver log on CDH 5.7 swamped with global.memstore.upperLimit deprecated messages when Trafodion under high transactional load

["Carol Pearson (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TRAFODION-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Carol Pearson updated TRAFODION-2606:
-------------------------------------
    Priority: Critical  (was: Major)

> Regionserver log on CDH 5.7 swamped with global.memstore.upperLimit deprecated messages when Trafodion under high transactional load
> ------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: TRAFODION-2606
>                 URL: https://issues.apache.org/jira/browse/TRAFODION-2606
>             Project: Apache Trafodion
>          Issue Type: Bug
>          Components: dtm
>    Affects Versions: 2.2-incubating
>         Environment: Testing trafodion on AWS 4 node d2.2xlarge cluster with Trafodion 2.2 daily build, using CDH 5.7.  Ran a version of the OrderEntry benchmark to produce workload
>            Reporter: Carol Pearson
>            Priority: Critical
>
> Was running a basic Trafodion checkout to validate a recently installed instance, and saw huge growth in the regionserver's log file.  In under 5 hours, I was close to rolling a log, and after 10 hours, I was 122MB into the second log.  On a different cluster, it was days before the log rolled. So I investigated further.  
> At the top, I found the usual startup stuff, and messages as I loaded my test tables,
> Then came pages and pages and pages and pages of errors about a deprecated call:
> 2017-05-03 19:52:18,858 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
> How many in this 5 hour old instance? 
> grep " hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size" hbase-cmf-trafhbase-REGIONSERVER-glacier01.cluster.local.log.out | wc -l
> 1084218
> [trafodion@glacier01 hbase]$
> In other words, a lot of extra energy is spent logging in the hbase regionserver.  Over a million of them.
> The loads to set up my tables are not transactional, so I didn't see the same level of message generation.
> To reproduce the problem, run a transactional workload on HBase 1.2/CDH 5.7.  I use a version of the order entry benchmark.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)