[jira] [Resolved] (HBASE-27327) Class missing at runtime

["fengxianjing (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-27327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

fengxianjing resolved HBASE-27327.
----------------------------------
    Resolution: Not A Bug

> Class missing at runtime
> ------------------------
>
>                 Key: HBASE-27327
>                 URL: https://issues.apache.org/jira/browse/HBASE-27327
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 2.3.4
>            Reporter: fengxianjing
>            Priority: Critical
>
> We found that some class cannot be found after the regionserver has been running for a period of time(more than a month). And more than half of the machines in our cluster have this problem. Some are still running normally, but _/rs-status_ cannot be opened，and others have various problems(such as rit, replication failed, abort failed)
> Some exceptions are as follows:
> {code:java}
> java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl
> 	at org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:49)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780)
> 	at org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112)
> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
> 	at org.apache.hadoop.hbase.http.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:66)
> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
> 	at org.apache.hadoop.hbase.http.ClickjackingPreventionFilter.doFilter(ClickjackingPreventionFilter.java:52)
> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
> 	at org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1495)
> {code}
> {code:java}
> 2022-08-24 19:22:52,536 ERROR [RS_CLOSE_REGION-regionserver/xxxx:26020-1] regionserver.HRegionServer: ***** ABORTING region server 10.x.x.x,26020,1659357427101: Replay of WAL required. Forcing server shutdown *****
> org.apache.hadoop.hbase.DroppedSnapshotException: region: xxxx:xxxx,xxxxxx,1661298619920.943104cbcf4a74db9896fd4abd051411.
>         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2906)
>         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2575)
>         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2547)
>         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2538)
>         at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1652)
>         at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1591)
>         at org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:114)
>         at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/regionserver/querymatcher/DeleteTracker$DeleteResult
>         at org.apache.hadoop.hbase.regionserver.querymatcher.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:108)
>         at org.apache.hadoop.hbase.regionserver.querymatcher.ScanQueryMatcher.checkDeleted(ScanQueryMatcher.java:209)
>         at org.apache.hadoop.hbase.regionserver.querymatcher.MinorCompactionScanQueryMatcher.match(MinorCompactionScanQueryMatcher.java:54)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:627)
>         at org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:127)
>         at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:69)
>         at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:1067)
>         at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2442)
>         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2842)
>         ... 10 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.regionserver.querymatcher.DeleteTracker$DeleteResult
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 19 more
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)