You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Josh Elser (JIRA)" <ji...@apache.org> on 2019/07/16 20:26:00 UTC
[jira] [Created] (HBASE-22701) Better handle invalid local
directory for DynamicClassLoader
Josh Elser created HBASE-22701:
----------------------------------
Summary: Better handle invalid local directory for DynamicClassLoader
Key: HBASE-22701
URL: https://issues.apache.org/jira/browse/HBASE-22701
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Josh Elser
Assignee: Josh Elser
Fix For: 2.3.0, 2.2.1, 2.1.6
If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local") which is not writable to it, you will get some weird errors on the scan path. I just saw this (again?) with Phoenix.
Specifically, the first attempt to reference DynamicClassLoader (via ProtobufUtil), will result in an ExceptionInInitializationError because the unchecked exception coming out of DynamicClassLoader's constructor interrupts the loading of {{DynamicClassLoader.class}}.
{noformat}
2019-07-14 06:25:34,284 ERROR [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020] coprocessor.MetaDataEndpointImpl: dropTable failed
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ExceptionInInitializerError
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ExceptionInInitializerError
at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
... 7 more
Caused by: java.lang.RuntimeException: Failed to create local dir /hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
at org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
at org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicClassLoader.java:98)
at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.<clinit>(ProtobufUtil.java:260)
... 16 more
{noformat}
Every subsequent call will result in a NoClassDefFoundError, because we already tried to load DynamicClassLoader.class once and failed.
{noformat}
2019-07-14 06:25:34,380 ERROR [RpcServer.Metadata.Fifo.handler=2,queue=2,port=16020] coprocessor.MetaDataEndpointImpl: dropTable failed
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder
at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
... 7 more
{noformat}
The client gets an error about this, and would presumably know that something is amiss, but an operator wouldn't potentially see this on their own.
I see two options:
# We abort the RegionServer when the DynamicClassLoader fails to run
# We catch the exception and treat the DynamicClassLoader as disabled (same action as if you had set {{hbase.use.dynamic.jars=false}}).
I want to do #1 so that we don't propagate bogus configuration, but it feels a bit "harsh" to do that. I think #2 is the right solution with a big-fat-warning.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)