[jira] [Created] (KNOX-2954) Gateway metric name of service contain hbase rowkey led to full gc

["liang.feng (Jira)" <ji...@apache.org>]

liang.feng created KNOX-2954:
--------------------------------

             Summary: Gateway metric name of service contain hbase rowkey led to full gc
                 Key: KNOX-2954
                 URL: https://issues.apache.org/jira/browse/KNOX-2954
             Project: Apache Knox
          Issue Type: Bug
          Components: Metrics, Server
            Reporter: liang.feng
         Attachments: image-2023-09-01-16-40-35-876.png, image-2023-09-01-16-49-57-346.png, image-2023-09-01-16-55-30-378.png, image-2023-09-01-17-37-51-597.png, mat_heap_leak_suspects_result.png

Our Gateway Server frequently full gc in a few days. I dumped jvm heap and analyzed dump file with mat tool. I found some leak suspects about jmx metric, These leaks The leak occurs in class org.apache.knox.gateway.services.metrics.impl.DefaultMetricsService. 

This is the analysis result generated by MAT

!mat_heap_leak_suspects_result.png!

 

This is detail of problem Suspect 3

!image-2023-09-01-16-40-35-876.png!

 

I found that many metrics which starting with service contain hbase rowkey by using visualVM tool. These metrics were generted when I used get api to query data from hbase. I think that name of service metric should not contain rowkey.

 

This is an example of metric name

!image-2023-09-01-16-55-30-378.png!

 

The reason for this problem is that client and service metric both call the InstrUtils.getServiceResourcePath function to get resource path , but the parameters they use are different when build metric name. Suppose the request url is [https://localhost:8443/gateway/sandbox/hbase/demo/row9. |https://bd-hbase-dev02.zeus.lianjia.com:8443/gateway/sandbox/hbase/demo/row9]When building a client metric name, the parameter of the calling InstrUtils.getServiceResourcePath function is httpServletRequest.getPathInfo(), its value is /hbase/demo/row9. When building a service metric name, the parameter of the calling InstrUtils.getServiceResourcePath function is RequestLine.getUri(), its value is /demo/row9. The purpose of InstrUtils.getServiceResourcePath is to obtain the first two elements of path. So service metric name contains rowkey.

I think that we should only obtain the first elements of path, when building service metric name.

The effect after applying the  ,

!image-2023-09-01-17-37-51-597.png!

 

 

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)