You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by "caijialiang (Jira)" <ji...@apache.org> on 2023/05/08 01:57:00 UTC
[jira] [Updated] (RANGER-4201) Hbase master can't start due to ranger-hbase-plugin jersey jar class loading order
[ https://issues.apache.org/jira/browse/RANGER-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
caijialiang updated RANGER-4201:
--------------------------------
Description:
Issue description: HBase fails to start after enabling Ranger 2.4 plugin. Problem analysis is required.
environment：
linux centos7.4
hbase 2.4.13
ranger 2.4
jdk1.8
hadoop 3.3.4
zookeeper 3.5.9
Symptom: After integrating Ranger with HBase, HBase Master cannot start, and Region Server also crashes after running for a period of time. The same error message occurs during both processes.
{code:java}
[master/gs-server-13481:16000:becomeActiveMaster] coprocessor.CoprocessorHost: The coprocessor org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor threw java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError
at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)
at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175)
at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)
at com.sun.jersey.api.client.Client.init(Client.java:343)
at com.sun.jersey.api.client.Client.access$000(Client.java:119)
at com.sun.jersey.api.client.Client$1.f(Client.java:192)
at com.sun.jersey.api.client.Client$1.f(Client.java:188)
at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
at com.sun.jersey.api.client.Client.<init>(Client.java:188)
at com.sun.jersey.api.client.Client.<init>(Client.java:171)
at com.sun.jersey.api.client.Client.create(Client.java:683)
at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:228)
at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:193)
at org.apache.ranger.plugin.util.RangerRESTClient.get(RangerRESTClient.java:473)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRangerRolesDownloadResponse(RangerAdminRESTClient.java:1340)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1202)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167)
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183)
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123)
at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495)
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245)
at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:1120)
at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:160)
at org.apache.hadoop.hbase.coprocessor.BaseEnvironment.startup(BaseEnvironment.java:69)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.checkAndLoadInstance(CoprocessorHost.java:285)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:171)
at org.apache.hadoop.hbase.master.MasterCoprocessorHost.<init>(MasterCoprocessorHost.java:155)
at org.apache.hadoop.hbase.master.HMaster.initializeCoprocessorHost(HMaster.java:3870)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:902)
at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2175)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:520)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.glassfish.jersey.internal.RuntimeDelegateImpl
at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:130)
at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:97)
at javax.ws.rs.core.MediaType.valueOf(MediaType.java:172)
at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:65)
... 33 more
Caused by: java.lang.ClassNotFoundException: org.glassfish.jersey.internal.RuntimeDelegateImpl
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:87)
at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:185)
at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:112)
... 36 more {code}
Steps to reproduce:
* 1.After installing HBase 2.x and Ranger 2.4, configure HBase by modifying the hbase-site.xml file:
* Set hbase.coprocessor.master.classes to org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
* Set hbase.coprocessor.region.classes to org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
* Set hbase.coprocessor.regionserver.classes to org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
* Restart HBase and check the HBase Master logs to observe the exception and see the Master crash soon after
solution：delete /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar
related issues
https://issues.apache.org/jira/browse/HBASE-22029
https://issues.apache.org/jira/browse/HBASE-22052
!image-2023-04-27-16-11-05-561.png!
Problem Analysis:
Conclusion: Exception caused by class loading order
Analysis process: The call chain is roughly as follows: master start {-}> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start ... ->com.sun.jersey.api.client.Client.create ...{-}>com.sun.jersey.core.spi.factory.MessageBodyFactory.init ->com.sun.jersey.core.header.MediaTypes ->javax.ws.rs.core.MediaType.valueOf -> javax.ws.rs.ext.RuntimeDelegate.getInstance -> javax.ws.rs.ext.RuntimeDelegate.findDelegate
I have fully investigated this call chain and found that the root cause of the problem is that javax.ws.rs.ext.RuntimeDelegate in the javax.ws.rs-api-2.1.1.jar package in HBase did not find org.glassfish.jersey.internal.RuntimeDelegateImpl when calling findDelegate, which caused the error. Specifically, javax.ws.rs.ext.RuntimeDelegate first looks for the default JAXRS_RUNTIME_DELEGATE, which is "javax.ws.rs.ext.RuntimeDelegate". If it cannot be found, it will fallback and load JAXRS_DEFAULT_RUNTIME_DELEGATE, which is org.glassfish.jersey.internal.RuntimeDelegateImpl.
The method "findDelegate" in "javax.ws.rs.ext.RuntimeDelegate" is as follows:
{code:java}
public abstract class RuntimeDelegate {
public static final String JAXRS_RUNTIME_DELEGATE_PROPERTY = "javax.ws.rs.ext.RuntimeDelegate";
private static final String JAXRS_DEFAULT_RUNTIME_DELEGATE = "org.glassfish.jersey.internal.RuntimeDelegateImpl";
private static final Object RD_LOCK = new Object();
private static ReflectPermission suppressAccessChecksPermission = new ReflectPermission("suppressAccessChecks");
private static volatile RuntimeDelegate cachedDelegate; protected RuntimeDelegate() {
}
public static RuntimeDelegate getInstance() {
// Double-check idiom for lazy initialization of fields.
// Local variable is used to limit the number of more expensive accesses to a volatile field.
RuntimeDelegate result = cachedDelegate;
if (result == null) { // First check (no locking)
synchronized (RD_LOCK) {
result = cachedDelegate;
if (result == null) { // Second check (with locking)
cachedDelegate = result = findDelegate();
}
}
}
return result;
} private static RuntimeDelegate findDelegate() {
try {
Object delegate = FactoryFinder.find(
JAXRS_RUNTIME_DELEGATE_PROPERTY,
JAXRS_DEFAULT_RUNTIME_DELEGATE,
RuntimeDelegate.class);
if (!(delegate instanceof RuntimeDelegate)) {
Class pClass = RuntimeDelegate.class;
String classnameAsResource = pClass.getName().replace('.', '/') + ".class";
ClassLoader loader = pClass.getClassLoader();
if (loader == null) {
loader = ClassLoader.getSystemClassLoader();
}
URL targetTypeURL = loader.getResource(classnameAsResource);
throw new LinkageError("ClassCastException: attempting to cast"
+ delegate.getClass().getClassLoader().getResource(classnameAsResource)
+ " to " + targetTypeURL);
}
return (RuntimeDelegate) delegate;
} catch (Exception ex) {
throw new RuntimeException(ex);
}
}
{code}
When the "jersey-core-1.19.3.jar" file is removed from "/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/", HBase can start normally.
It is not clear which class, either "javax.ws.rs.ext.RuntimeDelegate" or "org.glassfish.jersey.internal.RuntimeDelegateImpl", is loaded by HBase at this point.
To determine which class is loaded and from which JAR file it is loaded, we can first remove "/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar" and start HBase normally. Then, we can use the Arthas tool ([https://github.com/alibaba/arthas]) to sniff and determine which class is loaded and from which JAR file it is loaded by the HBase JVM.
After starting HBase normally, run the Arthas startup script "sudo -u hbase -EH ./as.sh " attach to the HBase process. Then, execute "sc -d javax.ws.rs.ext.RuntimeDelegat" to check whether "javax.ws.rs.ext.RuntimeDelegate" is loaded.
result：
{code:java}
[arthas@1719]$ sc -d javax.ws.rs.ext.RuntimeDelegate
class-info com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
code-source /usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/jersey-core-1.19.jar
name com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name AbstractRuntimeDelegate
modifier abstract,public
annotation
interfaces
super-class +-javax.ws.rs.ext.RuntimeDelegate
+-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
+-sun.misc.Launcher$ExtClassLoader@436a4e4b
classLoaderHash 4fca772d class-info com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
code-source /usr/bigtop/3.2.0/usr/lib/hadoop/lib/jersey-server-1.19.jar
name com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name RuntimeDelegateImpl
modifier public
annotation
interfaces
super-class +-com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
+-javax.ws.rs.ext.RuntimeDelegate
+-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
+-sun.misc.Launcher$ExtClassLoader@436a4e4b
classLoaderHash 4fca772d class-info javax.ws.rs.ext.RuntimeDelegate
code-source /usr/bigtop/3.2.0/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar
name javax.ws.rs.ext.RuntimeDelegate
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name RuntimeDelegate
modifier abstract,public
annotation
interfaces
super-class +-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
+-sun.misc.Launcher$ExtClassLoader@436a4e4b
classLoaderHash 4fca772dAffect(row-cnt:3) cost in 112 ms. {code}
By removing the jersey-core-1.19.3.jar file from /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/, we can see that the abstract class javax.ws.rs.ext.RuntimeDelegate is loaded by the AppClassLoader from the javax.ws.rs-api-2.1.1.jar file located in /usr/bigtop/3.2.0/usr/lib/hbase/lib/.
At the same time, the implementation of javax.ws.rs.ext.RuntimeDelegate, which is com.sun.jersey.server.impl.provider.RuntimeDelegateImpl, is loaded from the jersey-server-1.19.jar file located in /usr/bigtop/3.2.0/usr/lib/hadoop/lib/. Note that the hadoop lib directory is appended to HBase's classpath when HBase is started, which is why it can be loaded.
Additionally, the abstract class com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate, which inherits from RuntimeDelegate, is loaded from the jersey-core-1.19.jar file located in /usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/.
However, even after removing the jersey-core-1.19.3.jar file, we can still see that HBase loads the same version of jersey-core-1.19.jar from the hadoop lib directory. Therefore, the root cause of the error is not a class not found due to a version conflict, but rather due to the order in which the class loader loads the classes.
To verify this conclusion, the simplest way is to:
# Before deleting the jersey-core-1.19.3.jar file from /ranger-hbase-plugin-impl/, print out the loading information of all the classes involved in the stack trace of the heap dump exception in the log.
# After deleting the jersey-core-1.19.3.jar file, print out the loading information of all the classes involved in the stack trace of the heap dump exception in the log.
# Use regular expressions to batch delete the timestamps from the log, and then compare the two log files to see any differences in the class loading information.
In the Ranger 2.4 source code, Ranger implements its own class loader, RangerPluginClassLoader, which inherits from URLClassLoader. During initialization, it is passed the path to the ranger-\{componentname}{-}plugin-impl directory, and all the classes in this directory will be loaded by RangerPluginClassLoader. The parent is set to null, which in my understanding, is to block delegation loading. This is because all the classes in the ranger{-}
{componentname}
-plugin-impl directory are dependencies of the Ranger plugin. For example, if the version of a jar that it depends on is lower, if the delegation loading mechanism is used, it is easy to load the higher version of the class under the component/lib directory, leading to class conflict issues. Therefore, parent is set to null.
Internally, RangerPluginClassLoader first tries to load the jar files under the ranger-\{componentname}-plugin-impl directory using the parent URLClassLoader. If it can't be loaded, it uses the componentClassLoader, which is the current thread context's class loader, to load it.
!image-2023-04-27-16-13-44-504.png!
Next, you can take a look at the source code of the "RangerAuthorizationCoprocessor" class in Ranger. This class initializes an instance of "rangerPluginClassLoader" and activates it. Within this class, there are pre and post hooks for various HBase operations, primarily for performing range permission-related operations. This triggers the loading of Ranger-related classes by the rangerPluginClassLoader.
!image-2023-04-27-16-14-10-803.png!
Why does the hook activate the Ranger class loader at the beginning and de-activate it after the hook method ends? Doesn't this approach impact performance?
!image-2023-04-27-16-14-37-202.png!
The reason for de-activating the Ranger class loader after the hook method ends is because the rangerPluginClassLoader code sets itself as the current thread's context class loader. If it's not deactivated, it can cause conflicts with the lower version packages loaded in the .../ranger-hbase-plugin-impl/ directory, which can cause issues during subsequent operations. Therefore, the conclusion is that this involves the rangerPluginClassLoader and the AppClassLoader of the thread's context class loader.
We also need to understand how to load an instance of "RuntimeDelegate" and how it searches for the implementation of "javax.ws.rs.ext.RuntimeDelegate". We should also investigate why it cannot find the implementation class and instead looks for "org.glassfish.jersey.internal.RuntimeDelegateImpl", causing an error.
First, the "RuntimeDelegate" class is loaded via the "javax.ws.rs.ext.RuntimeDelegate.findDelegate" method, which calls "FactoryFinder.find" to load the class. Once loaded, the instance must be of type "RuntimeDelegate".
{code:java}
private static RuntimeDelegate findDelegate() {
try {
Object delegate = FactoryFinder.find("javax.ws.rs.ext.RuntimeDelegate", "org.glassfish.jersey.internal.RuntimeDelegateImpl", RuntimeDelegate.class);
if (!(delegate instanceof RuntimeDelegate)) {
Class<RuntimeDelegate> pClass = RuntimeDelegate.class;
String classnameAsResource = pClass.getName().replace('.', '/') + ".class";
ClassLoader loader = pClass.getClassLoader();
if (loader == null)
loader = ClassLoader.getSystemClassLoader();
URL targetTypeURL = loader.getResource(classnameAsResource);
throw new LinkageError("ClassCastException: attempting to cast" + delegate
.getClass().getClassLoader().getResource(classnameAsResource) + " to " + targetTypeURL);
}
return (RuntimeDelegate)delegate;
} catch (Exception ex) {
throw new RuntimeException(ex);
}
} {code}
The process of loading the class via "FactoryFinder.find" is contained in the "javax.ws.rs.ext.FactoryFinder.find" method, as shown in the following code:
{code:java}
/**
* Finds the implementation {@code Class} for the given factory name,
* or if that fails, finds the {@code Class} for the given fallback
* class name and create its instance. The arguments supplied MUST be
* used in order. If using the first argument is successful, the second
* one will not be used.
* <p>
* This method is package private so that this code can be shared.
*
* @param factoryId the name of the factory to find, which is
* a system property.
* @param fallbackClassName the implementation class name, which is
* to be used only if nothing else.
* is found; {@code null} to indicate that
* there is no fallback class name.
* @param service service to be found.
* @param <T> type of the service to be found.
* @return the instance of the specified service; may not be {@code null}.
* @throws ClassNotFoundException if the given class could not be found
* or could not be instantiated.
*/
static <T> Object find(final String factoryId, final String fallbackClassName, Class<T> service) throws ClassNotFoundException {
ClassLoader classLoader = getContextClassLoader(); try {
Iterator<T> iterator = ServiceLoader.load(service, FactoryFinder.getContextClassLoader()).iterator(); if(iterator.hasNext()) {
return iterator.next();
}
} catch (Exception | ServiceConfigurationError ex) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId + ".", ex);
} try {
Iterator<T> iterator = ServiceLoader.load(service, FactoryFinder.class.getClassLoader()).iterator(); if(iterator.hasNext()) {
return iterator.next();
}
} catch (Exception | ServiceConfigurationError ex) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId + ".", ex);
} // try to read from $java.home/lib/jaxrs.properties
FileInputStream inputStream = null;
String configFile = null;
try {
String javah = System.getProperty("java.home");
configFile = javah + File.separator + "lib" + File.separator + "jaxrs.properties";
File f = new File(configFile);
if (f.exists()) {
Properties props = new Properties();
inputStream = new FileInputStream(f);
props.load(inputStream);
String factoryClassName = props.getProperty(factoryId);
return newInstance(factoryClassName, classLoader);
}
} catch (Exception ex) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId
+ " from $java.home/lib/jaxrs.properties", ex);
} finally {
if (inputStream != null) {
try {
inputStream.close();
} catch (IOException ex) {
LOGGER.log(Level.FINER, String.format("Error closing %s file.", configFile), ex);
}
}
} // Use the system property
try {
String systemProp = System.getProperty(factoryId);
if (systemProp != null) {
return newInstance(systemProp, classLoader);
}
} catch (SecurityException se) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId
+ " from a system property", se);
} if (fallbackClassName == null) {
throw new ClassNotFoundException(
"Provider for " + factoryId + " cannot be found", null);
} return newInstance(fallbackClassName, classLoader);
}
} {code}
To summarize the "FactoryFinder.find" method, it uses the thread context class loader to load "javax.ws.rs.ext.RuntimeDelegate". If that fails, it attempts to load it from system properties or alternative class names. If it still cannot be found, it tries to load "org.glassfish.jersey.internal.RuntimeDelegateImpl". If it cannot be loaded, it throws an exception.
When the /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar file was not deleted, the class loading order was as follows:
javax.ws.rs.ext.RuntimeDelegate(appClassLoader) -> javax.ws.rs.ext.RuntimeDelegate.HeaderDelegate(RangerPluginClassLoader) -> javax.ws.rs.ext.RuntimeDelegate.HeaderDelegateProvider(RangerPluginClassLoader jersey-core-1.19.3.jar ranger-hbase-plugin-impl/jersey-core-1.19.3.jar).
The RangerPluginClassLoader searched for the com.sun.jersey.spi.HeaderDelegateProvider implementation class in the META-INF/services/com.sun.jersey.spi.HeaderDelegateProvider file of the jersey-core-1.19.3.jar file, found the com.sun.jersey.core.impl.provider.header.LocaleProvider class in ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, and returned it.
However, javax.ws.rs.ext.RuntimeDelegate.findDelegate would fail because delegate instanceof RuntimeDelegate returned false.
As a fallback, it searched for org.glassfish.jersey.internal.RuntimeDelegateImpl, which was not found in the hbase classpath and caused an exception to be thrown.
After removing ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, HBase can be started normally. The class loading order is as follows:
javax.ws.rs.ext.RuntimeDelegate is loaded from /usr/lib/hadoop/lib/javax.ws.rs-api-2.1.1.jar. Then, the implementation class of javax.ws.rs.ext.RuntimeDelegate is searched for, which is found in /usr/lib/hadoop/lib/jersey-server-1.19.jar->META-INF/services/javax.ws.rs.ext.RuntimeDelegate->com.sun.jersey.server.impl.provider.RuntimeDelegateImpl, and successfully loaded.
The above class loading order was determined by modifying the code in the RangerPluginClassLoader to print out the process of loading all classes.
!image-2023-04-27-16-15-44-539.png!
The following is a log that includes the class loading order. By analyzing this log in conjunction with the Ranger class loader code, the above results can be determined.
[^hbase-hbase-master.err.log]
[^hbase-hbase-master.log]
was:
Issue description: HBase fails to start after enabling Ranger 2.4 plugin. Problem analysis is required.
environment：
linux centos7.4
hbase 2.4.13
ranger 2.4
jdk1.8
hadoop 3.3.4
zookeeper 3.5.9
Symptom: After integrating Ranger with HBase, HBase Master cannot start, and Region Server also crashes after running for a period of time. The same error message occurs during both processes.
{code:java}
[master/gs-server-13481:16000:becomeActiveMaster] coprocessor.CoprocessorHost: The coprocessor org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor threw java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError
at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)
at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175)
at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)
at com.sun.jersey.api.client.Client.init(Client.java:343)
at com.sun.jersey.api.client.Client.access$000(Client.java:119)
at com.sun.jersey.api.client.Client$1.f(Client.java:192)
at com.sun.jersey.api.client.Client$1.f(Client.java:188)
at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
at com.sun.jersey.api.client.Client.<init>(Client.java:188)
at com.sun.jersey.api.client.Client.<init>(Client.java:171)
at com.sun.jersey.api.client.Client.create(Client.java:683)
at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:228)
at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:193)
at org.apache.ranger.plugin.util.RangerRESTClient.get(RangerRESTClient.java:473)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRangerRolesDownloadResponse(RangerAdminRESTClient.java:1340)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1202)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167)
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183)
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123)
at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495)
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245)
at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:1120)
at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:160)
at org.apache.hadoop.hbase.coprocessor.BaseEnvironment.startup(BaseEnvironment.java:69)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.checkAndLoadInstance(CoprocessorHost.java:285)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:171)
at org.apache.hadoop.hbase.master.MasterCoprocessorHost.<init>(MasterCoprocessorHost.java:155)
at org.apache.hadoop.hbase.master.HMaster.initializeCoprocessorHost(HMaster.java:3870)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:902)
at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2175)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:520)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.glassfish.jersey.internal.RuntimeDelegateImpl
at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:130)
at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:97)
at javax.ws.rs.core.MediaType.valueOf(MediaType.java:172)
at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:65)
... 33 more
Caused by: java.lang.ClassNotFoundException: org.glassfish.jersey.internal.RuntimeDelegateImpl
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:87)
at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:185)
at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:112)
... 36 more {code}
Steps to reproduce:
* 1.After installing HBase 2.x and Ranger 2.4, configure HBase by modifying the hbase-site.xml file:
* Set hbase.coprocessor.master.classes to org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
* Set hbase.coprocessor.region.classes to org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
* Set hbase.coprocessor.regionserver.classes to org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
* Restart HBase and check the HBase Master logs to observe the exception and see the Master crash soon after
solution：delete /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar
related issues
https://issues.apache.org/jira/browse/HBASE-22029
https://issues.apache.org/jira/browse/HBASE-22052
!image-2023-04-27-16-11-05-561.png!
Problem Analysis:
Conclusion: Exception caused by class loading order
Analysis process: The call chain is roughly as follows: master start {-}> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start ... ->com.sun.jersey.api.client.Client.create ...{-}>com.sun.jersey.core.spi.factory.MessageBodyFactory.init ->com.sun.jersey.core.header.MediaTypes ->javax.ws.rs.core.MediaType.valueOf -> javax.ws.rs.ext.RuntimeDelegate.getInstance -> javax.ws.rs.ext.RuntimeDelegate.findDelegate
I have fully investigated this call chain and found that the root cause of the problem is that javax.ws.rs.ext.RuntimeDelegate in the javax.ws.rs-api-2.1.1.jar package in HBase did not find org.glassfish.jersey.internal.RuntimeDelegateImpl when calling findDelegate, which caused the error. Specifically, javax.ws.rs.ext.RuntimeDelegate first looks for the default JAXRS_RUNTIME_DELEGATE, which is "javax.ws.rs.ext.RuntimeDelegate". If it cannot be found, it will fallback and load JAXRS_DEFAULT_RUNTIME_DELEGATE, which is org.glassfish.jersey.internal.RuntimeDelegateImpl.
The method "findDelegate" in "javax.ws.rs.ext.RuntimeDelegate" is as follows:
{code:java}
public abstract class RuntimeDelegate {
public static final String JAXRS_RUNTIME_DELEGATE_PROPERTY = "javax.ws.rs.ext.RuntimeDelegate";
private static final String JAXRS_DEFAULT_RUNTIME_DELEGATE = "org.glassfish.jersey.internal.RuntimeDelegateImpl";
private static final Object RD_LOCK = new Object();
private static ReflectPermission suppressAccessChecksPermission = new ReflectPermission("suppressAccessChecks");
private static volatile RuntimeDelegate cachedDelegate; protected RuntimeDelegate() {
}
public static RuntimeDelegate getInstance() {
// Double-check idiom for lazy initialization of fields.
// Local variable is used to limit the number of more expensive accesses to a volatile field.
RuntimeDelegate result = cachedDelegate;
if (result == null) { // First check (no locking)
synchronized (RD_LOCK) {
result = cachedDelegate;
if (result == null) { // Second check (with locking)
cachedDelegate = result = findDelegate();
}
}
}
return result;
} private static RuntimeDelegate findDelegate() {
try {
Object delegate = FactoryFinder.find(
JAXRS_RUNTIME_DELEGATE_PROPERTY,
JAXRS_DEFAULT_RUNTIME_DELEGATE,
RuntimeDelegate.class);
if (!(delegate instanceof RuntimeDelegate)) {
Class pClass = RuntimeDelegate.class;
String classnameAsResource = pClass.getName().replace('.', '/') + ".class";
ClassLoader loader = pClass.getClassLoader();
if (loader == null) {
loader = ClassLoader.getSystemClassLoader();
}
URL targetTypeURL = loader.getResource(classnameAsResource);
throw new LinkageError("ClassCastException: attempting to cast"
+ delegate.getClass().getClassLoader().getResource(classnameAsResource)
+ " to " + targetTypeURL);
}
return (RuntimeDelegate) delegate;
} catch (Exception ex) {
throw new RuntimeException(ex);
}
}
{code}
When the "jersey-core-1.19.3.jar" file is removed from "/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/", HBase can start normally.
It is not clear which class, either "javax.ws.rs.ext.RuntimeDelegate" or "org.glassfish.jersey.internal.RuntimeDelegateImpl", is loaded by HBase at this point.
To determine which class is loaded and from which JAR file it is loaded, we can first remove "/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar" and start HBase normally. Then, we can use the Arthas tool ([https://github.com/alibaba/arthas]) to sniff and determine which class is loaded and from which JAR file it is loaded by the HBase JVM.
After starting HBase normally, run the Arthas startup script "sudo -u hbase -EH ./as.sh " attach to the HBase process. Then, execute "sc -d javax.ws.rs.ext.RuntimeDelegat" to check whether "javax.ws.rs.ext.RuntimeDelegate" is loaded.
result：
{code:java}
[arthas@1719]$ sc -d javax.ws.rs.ext.RuntimeDelegate
class-info com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
code-source /usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/jersey-core-1.19.jar
name com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name AbstractRuntimeDelegate
modifier abstract,public
annotation
interfaces
super-class +-javax.ws.rs.ext.RuntimeDelegate
+-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
+-sun.misc.Launcher$ExtClassLoader@436a4e4b
classLoaderHash 4fca772d class-info com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
code-source /usr/bigtop/3.2.0/usr/lib/hadoop/lib/jersey-server-1.19.jar
name com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name RuntimeDelegateImpl
modifier public
annotation
interfaces
super-class +-com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
+-javax.ws.rs.ext.RuntimeDelegate
+-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
+-sun.misc.Launcher$ExtClassLoader@436a4e4b
classLoaderHash 4fca772d class-info javax.ws.rs.ext.RuntimeDelegate
code-source /usr/bigtop/3.2.0/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar
name javax.ws.rs.ext.RuntimeDelegate
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name RuntimeDelegate
modifier abstract,public
annotation
interfaces
super-class +-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
+-sun.misc.Launcher$ExtClassLoader@436a4e4b
classLoaderHash 4fca772dAffect(row-cnt:3) cost in 112 ms. {code}
By removing the jersey-core-1.19.3.jar file from /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/, we can see that the abstract class javax.ws.rs.ext.RuntimeDelegate is loaded by the AppClassLoader from the javax.ws.rs-api-2.1.1.jar file located in /usr/bigtop/3.2.0/usr/lib/hbase/lib/.
At the same time, the implementation of javax.ws.rs.ext.RuntimeDelegate, which is com.sun.jersey.server.impl.provider.RuntimeDelegateImpl, is loaded from the jersey-server-1.19.jar file located in /usr/bigtop/3.2.0/usr/lib/hadoop/lib/. Note that the hadoop lib directory is appended to HBase's classpath when HBase is started, which is why it can be loaded.
Additionally, the abstract class com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate, which inherits from RuntimeDelegate, is loaded from the jersey-core-1.19.jar file located in /usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/.
However, even after removing the jersey-core-1.19.3.jar file, we can still see that HBase loads the same version of jersey-core-1.19.jar from the hadoop lib directory. Therefore, the root cause of the error is not a class not found due to a version conflict, but rather due to the order in which the class loader loads the classes.
To verify this conclusion, the simplest way is to:
# Before deleting the jersey-core-1.19.3.jar file from /ranger-hbase-plugin-impl/, print out the loading information of all the classes involved in the stack trace of the heap dump exception in the log.
# After deleting the jersey-core-1.19.3.jar file, print out the loading information of all the classes involved in the stack trace of the heap dump exception in the log.
# Use regular expressions to batch delete the timestamps from the log, and then compare the two log files to see any differences in the class loading information.
In the Ranger 2.4 source code, Ranger implements its own class loader, RangerPluginClassLoader, which inherits from URLClassLoader. During initialization, it is passed the path to the ranger-\{componentname}{-}plugin-impl directory, and all the classes in this directory will be loaded by RangerPluginClassLoader. The parent is set to null, which in my understanding, is to block delegation loading. This is because all the classes in the ranger{-}{componentname}-plugin-impl directory are dependencies of the Ranger plugin. For example, if the version of a jar that it depends on is lower, if the delegation loading mechanism is used, it is easy to load the higher version of the class under the component/lib directory, leading to class conflict issues. Therefore, parent is set to null.
Internally, RangerPluginClassLoader first tries to load the jar files under the ranger-\{componentname}-plugin-impl directory using the parent URLClassLoader. If it can't be loaded, it uses the componentClassLoader, which is the current thread context's class loader, to load it.
!image-2023-04-27-16-13-44-504.png!
Next, you can take a look at the source code of the "RangerAuthorizationCoprocessor" class in Ranger. This class initializes an instance of "rangerPluginClassLoader" and activates it. Within this class, there are pre and post hooks for various HBase operations, primarily for performing range permission-related operations. This triggers the loading of Ranger-related classes by the rangerPluginClassLoader.
!image-2023-04-27-16-14-10-803.png!
Why does the hook activate the Ranger class loader at the beginning and de-activate it after the hook method ends? Doesn't this approach impact performance?
!image-2023-04-27-16-14-37-202.png!
The reason for de-activating the Ranger class loader after the hook method ends is because the rangerPluginClassLoader code sets itself as the current thread's context class loader. If it's not deactivated, it can cause conflicts with the lower version packages loaded in the .../ranger-hbase-plugin-impl/ directory, which can cause issues during subsequent operations. Therefore, the conclusion is that this involves the rangerPluginClassLoader and the AppClassLoader of the thread's context class loader.
We also need to understand how to load an instance of "RuntimeDelegate" and how it searches for the implementation of "javax.ws.rs.ext.RuntimeDelegate". We should also investigate why it cannot find the implementation class and instead looks for "org.glassfish.jersey.internal.RuntimeDelegateImpl", causing an error.
First, the "RuntimeDelegate" class is loaded via the "javax.ws.rs.ext.RuntimeDelegate.findDelegate" method, which calls "FactoryFinder.find" to load the class. Once loaded, the instance must be of type "RuntimeDelegate".
{code:java}
private static RuntimeDelegate findDelegate() {
try {
Object delegate = FactoryFinder.find("javax.ws.rs.ext.RuntimeDelegate", "org.glassfish.jersey.internal.RuntimeDelegateImpl", RuntimeDelegate.class);
if (!(delegate instanceof RuntimeDelegate)) {
Class<RuntimeDelegate> pClass = RuntimeDelegate.class;
String classnameAsResource = pClass.getName().replace('.', '/') + ".class";
ClassLoader loader = pClass.getClassLoader();
if (loader == null)
loader = ClassLoader.getSystemClassLoader();
URL targetTypeURL = loader.getResource(classnameAsResource);
throw new LinkageError("ClassCastException: attempting to cast" + delegate
.getClass().getClassLoader().getResource(classnameAsResource) + " to " + targetTypeURL);
}
return (RuntimeDelegate)delegate;
} catch (Exception ex) {
throw new RuntimeException(ex);
}
} {code}
The process of loading the class via "FactoryFinder.find" is contained in the "javax.ws.rs.ext.FactoryFinder.find" method, as shown in the following code:
{code:java}
/**
* Finds the implementation {@code Class} for the given factory name,
* or if that fails, finds the {@code Class} for the given fallback
* class name and create its instance. The arguments supplied MUST be
* used in order. If using the first argument is successful, the second
* one will not be used.
* <p>
* This method is package private so that this code can be shared.
*
* @param factoryId the name of the factory to find, which is
* a system property.
* @param fallbackClassName the implementation class name, which is
* to be used only if nothing else.
* is found; {@code null} to indicate that
* there is no fallback class name.
* @param service service to be found.
* @param <T> type of the service to be found.
* @return the instance of the specified service; may not be {@code null}.
* @throws ClassNotFoundException if the given class could not be found
* or could not be instantiated.
*/
static <T> Object find(final String factoryId, final String fallbackClassName, Class<T> service) throws ClassNotFoundException {
ClassLoader classLoader = getContextClassLoader(); try {
Iterator<T> iterator = ServiceLoader.load(service, FactoryFinder.getContextClassLoader()).iterator(); if(iterator.hasNext()) {
return iterator.next();
}
} catch (Exception | ServiceConfigurationError ex) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId + ".", ex);
} try {
Iterator<T> iterator = ServiceLoader.load(service, FactoryFinder.class.getClassLoader()).iterator(); if(iterator.hasNext()) {
return iterator.next();
}
} catch (Exception | ServiceConfigurationError ex) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId + ".", ex);
} // try to read from $java.home/lib/jaxrs.properties
FileInputStream inputStream = null;
String configFile = null;
try {
String javah = System.getProperty("java.home");
configFile = javah + File.separator + "lib" + File.separator + "jaxrs.properties";
File f = new File(configFile);
if (f.exists()) {
Properties props = new Properties();
inputStream = new FileInputStream(f);
props.load(inputStream);
String factoryClassName = props.getProperty(factoryId);
return newInstance(factoryClassName, classLoader);
}
} catch (Exception ex) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId
+ " from $java.home/lib/jaxrs.properties", ex);
} finally {
if (inputStream != null) {
try {
inputStream.close();
} catch (IOException ex) {
LOGGER.log(Level.FINER, String.format("Error closing %s file.", configFile), ex);
}
}
} // Use the system property
try {
String systemProp = System.getProperty(factoryId);
if (systemProp != null) {
return newInstance(systemProp, classLoader);
}
} catch (SecurityException se) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId
+ " from a system property", se);
} if (fallbackClassName == null) {
throw new ClassNotFoundException(
"Provider for " + factoryId + " cannot be found", null);
} return newInstance(fallbackClassName, classLoader);
}
} {code}
To summarize the "FactoryFinder.find" method, it uses the thread context class loader to load "javax.ws.rs.ext.RuntimeDelegate". If that fails, it attempts to load it from system properties or alternative class names. If it still cannot be found, it tries to load "org.glassfish.jersey.internal.RuntimeDelegateImpl". If it cannot be loaded, it throws an exception.
When the /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar file was not deleted, the class loading order was as follows:
javax.ws.rs.ext.RuntimeDelegate(appClassLoader) -> javax.ws.rs.ext.RuntimeDelegate.HeaderDelegate(RangerPluginClassLoader) -> javax.ws.rs.ext.RuntimeDelegate.HeaderDelegateProvider(RangerPluginClassLoader jersey-core-1.19.3.jar ranger-hbase-plugin-impl/jersey-core-1.19.3.jar).
The RangerPluginClassLoader searched for the com.sun.jersey.spi.HeaderDelegateProvider implementation class in the META-INF/services/com.sun.jersey.spi.HeaderDelegateProvider file of the jersey-core-1.19.3.jar file, found the com.sun.jersey.core.impl.provider.header.LocaleProvider class in ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, and returned it.
However, javax.ws.rs.ext.RuntimeDelegate.findDelegate would fail because delegate instanceof RuntimeDelegate returned false.
As a fallback, it searched for org.glassfish.jersey.internal.RuntimeDelegateImpl, which was not found in the hbase classpath and caused an exception to be thrown.
After removing ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, HBase can be started normally. The class loading order is as follows:
javax.ws.rs.ext.RuntimeDelegate is loaded from /usr/lib/hadoop/lib/javax.ws.rs-api-2.1.1.jar. Then, the implementation class of javax.ws.rs.ext.RuntimeDelegate is searched for, which is found in /usr/lib/hadoop/lib/jersey-server-1.19.jar->META-INF/services/javax.ws.rs.ext.RuntimeDelegate->com.sun.jersey.server.impl.provider.RuntimeDelegateImpl, and successfully loaded.
The above class loading order was determined by modifying the code in the RangerPluginClassLoader to print out the process of loading all classes.
!image-2023-04-27-16-15-44-539.png!
The following is a log that includes the class loading order. By analyzing this log in conjunction with the Ranger class loader code, the above results can be determined.
[^hbase-hbase-master.err.log]
[^hbase-hbase-master.log]
> Hbase master can't start due to ranger-hbase-plugin jersey jar class loading order
> -----------------------------------------------------------------------------------
>
> Key: RANGER-4201
> URL: https://issues.apache.org/jira/browse/RANGER-4201
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
> Affects Versions: 2.3.0, 2.4.0
> Reporter: caijialiang
> Assignee: caijialiang
> Priority: Major
> Attachments: hbase-hbase-master.err.log, hbase-hbase-master.log, image-2023-04-27-16-11-05-561.png, image-2023-04-27-16-13-44-504.png, image-2023-04-27-16-14-10-803.png, image-2023-04-27-16-14-37-202.png, image-2023-04-27-16-15-44-539.png, patch-RANGER-4201.diff
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
>
> Issue description: HBase fails to start after enabling Ranger 2.4 plugin. Problem analysis is required.
> environment：
> linux centos7.4
> hbase 2.4.13
> ranger 2.4
> jdk1.8
> hadoop 3.3.4
> zookeeper 3.5.9
> Symptom: After integrating Ranger with HBase, HBase Master cannot start, and Region Server also crashes after running for a period of time. The same error message occurs during both processes.
>
> {code:java}
> [master/gs-server-13481:16000:becomeActiveMaster] coprocessor.CoprocessorHost: The coprocessor org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor threw java.lang.ExceptionInInitializerError
> java.lang.ExceptionInInitializerError
> at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)
> at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175)
> at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)
> at com.sun.jersey.api.client.Client.init(Client.java:343)
> at com.sun.jersey.api.client.Client.access$000(Client.java:119)
> at com.sun.jersey.api.client.Client$1.f(Client.java:192)
> at com.sun.jersey.api.client.Client$1.f(Client.java:188)
> at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
> at com.sun.jersey.api.client.Client.<init>(Client.java:188)
> at com.sun.jersey.api.client.Client.<init>(Client.java:171)
> at com.sun.jersey.api.client.Client.create(Client.java:683)
> at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:228)
> at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:193)
> at org.apache.ranger.plugin.util.RangerRESTClient.get(RangerRESTClient.java:473)
> at org.apache.ranger.admin.client.RangerAdminRESTClient.getRangerRolesDownloadResponse(RangerAdminRESTClient.java:1340)
> at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1202)
> at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167)
> at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183)
> at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123)
> at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495)
> at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144)
> at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245)
> at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:1120)
> at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:160)
> at org.apache.hadoop.hbase.coprocessor.BaseEnvironment.startup(BaseEnvironment.java:69)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.checkAndLoadInstance(CoprocessorHost.java:285)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:171)
> at org.apache.hadoop.hbase.master.MasterCoprocessorHost.<init>(MasterCoprocessorHost.java:155)
> at org.apache.hadoop.hbase.master.HMaster.initializeCoprocessorHost(HMaster.java:3870)
> at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:902)
> at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2175)
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:520)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.glassfish.jersey.internal.RuntimeDelegateImpl
> at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:130)
> at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:97)
> at javax.ws.rs.core.MediaType.valueOf(MediaType.java:172)
> at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:65)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.glassfish.jersey.internal.RuntimeDelegateImpl
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:87)
> at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:185)
> at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:112)
> ... 36 more {code}
> Steps to reproduce:
>
> * 1.After installing HBase 2.x and Ranger 2.4, configure HBase by modifying the hbase-site.xml file:
> * Set hbase.coprocessor.master.classes to org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
> * Set hbase.coprocessor.region.classes to org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
> * Set hbase.coprocessor.regionserver.classes to org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
> * Restart HBase and check the HBase Master logs to observe the exception and see the Master crash soon after
> solution：delete /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar
>
> related issues
> https://issues.apache.org/jira/browse/HBASE-22029
> https://issues.apache.org/jira/browse/HBASE-22052
>
> !image-2023-04-27-16-11-05-561.png!
> Problem Analysis:
> Conclusion: Exception caused by class loading order
>
> Analysis process: The call chain is roughly as follows: master start {-}> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start ... ->com.sun.jersey.api.client.Client.create ...{-}>com.sun.jersey.core.spi.factory.MessageBodyFactory.init ->com.sun.jersey.core.header.MediaTypes ->javax.ws.rs.core.MediaType.valueOf -> javax.ws.rs.ext.RuntimeDelegate.getInstance -> javax.ws.rs.ext.RuntimeDelegate.findDelegate
> I have fully investigated this call chain and found that the root cause of the problem is that javax.ws.rs.ext.RuntimeDelegate in the javax.ws.rs-api-2.1.1.jar package in HBase did not find org.glassfish.jersey.internal.RuntimeDelegateImpl when calling findDelegate, which caused the error. Specifically, javax.ws.rs.ext.RuntimeDelegate first looks for the default JAXRS_RUNTIME_DELEGATE, which is "javax.ws.rs.ext.RuntimeDelegate". If it cannot be found, it will fallback and load JAXRS_DEFAULT_RUNTIME_DELEGATE, which is org.glassfish.jersey.internal.RuntimeDelegateImpl.
> The method "findDelegate" in "javax.ws.rs.ext.RuntimeDelegate" is as follows:
> {code:java}
> public abstract class RuntimeDelegate {
> public static final String JAXRS_RUNTIME_DELEGATE_PROPERTY = "javax.ws.rs.ext.RuntimeDelegate";
> private static final String JAXRS_DEFAULT_RUNTIME_DELEGATE = "org.glassfish.jersey.internal.RuntimeDelegateImpl";
> private static final Object RD_LOCK = new Object();
> private static ReflectPermission suppressAccessChecksPermission = new ReflectPermission("suppressAccessChecks");
> private static volatile RuntimeDelegate cachedDelegate; protected RuntimeDelegate() {
> }
> public static RuntimeDelegate getInstance() {
> // Double-check idiom for lazy initialization of fields.
> // Local variable is used to limit the number of more expensive accesses to a volatile field.
> RuntimeDelegate result = cachedDelegate;
> if (result == null) { // First check (no locking)
> synchronized (RD_LOCK) {
> result = cachedDelegate;
> if (result == null) { // Second check (with locking)
> cachedDelegate = result = findDelegate();
> }
> }
> }
> return result;
> } private static RuntimeDelegate findDelegate() {
> try {
> Object delegate = FactoryFinder.find(
> JAXRS_RUNTIME_DELEGATE_PROPERTY,
> JAXRS_DEFAULT_RUNTIME_DELEGATE,
> RuntimeDelegate.class);
> if (!(delegate instanceof RuntimeDelegate)) {
> Class pClass = RuntimeDelegate.class;
> String classnameAsResource = pClass.getName().replace('.', '/') + ".class";
> ClassLoader loader = pClass.getClassLoader();
> if (loader == null) {
> loader = ClassLoader.getSystemClassLoader();
> }
> URL targetTypeURL = loader.getResource(classnameAsResource);
> throw new LinkageError("ClassCastException: attempting to cast"
> + delegate.getClass().getClassLoader().getResource(classnameAsResource)
> + " to " + targetTypeURL);
> }
> return (RuntimeDelegate) delegate;
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> }
> {code}
> When the "jersey-core-1.19.3.jar" file is removed from "/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/", HBase can start normally.
>
> It is not clear which class, either "javax.ws.rs.ext.RuntimeDelegate" or "org.glassfish.jersey.internal.RuntimeDelegateImpl", is loaded by HBase at this point.
>
> To determine which class is loaded and from which JAR file it is loaded, we can first remove "/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar" and start HBase normally. Then, we can use the Arthas tool ([https://github.com/alibaba/arthas]) to sniff and determine which class is loaded and from which JAR file it is loaded by the HBase JVM.
> After starting HBase normally, run the Arthas startup script "sudo -u hbase -EH ./as.sh " attach to the HBase process. Then, execute "sc -d javax.ws.rs.ext.RuntimeDelegat" to check whether "javax.ws.rs.ext.RuntimeDelegate" is loaded.
>
> result：
> {code:java}
> [arthas@1719]$ sc -d javax.ws.rs.ext.RuntimeDelegate
> class-info com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
> code-source /usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/jersey-core-1.19.jar
> name com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
> isInterface false
> isAnnotation false
> isEnum false
> isAnonymousClass false
> isArray false
> isLocalClass false
> isMemberClass false
> isPrimitive false
> isSynthetic false
> simple-name AbstractRuntimeDelegate
> modifier abstract,public
> annotation
> interfaces
> super-class +-javax.ws.rs.ext.RuntimeDelegate
> +-java.lang.Object
> class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
> +-sun.misc.Launcher$ExtClassLoader@436a4e4b
> classLoaderHash 4fca772d class-info com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
> code-source /usr/bigtop/3.2.0/usr/lib/hadoop/lib/jersey-server-1.19.jar
> name com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
> isInterface false
> isAnnotation false
> isEnum false
> isAnonymousClass false
> isArray false
> isLocalClass false
> isMemberClass false
> isPrimitive false
> isSynthetic false
> simple-name RuntimeDelegateImpl
> modifier public
> annotation
> interfaces
> super-class +-com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
> +-javax.ws.rs.ext.RuntimeDelegate
> +-java.lang.Object
> class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
> +-sun.misc.Launcher$ExtClassLoader@436a4e4b
> classLoaderHash 4fca772d class-info javax.ws.rs.ext.RuntimeDelegate
> code-source /usr/bigtop/3.2.0/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar
> name javax.ws.rs.ext.RuntimeDelegate
> isInterface false
> isAnnotation false
> isEnum false
> isAnonymousClass false
> isArray false
> isLocalClass false
> isMemberClass false
> isPrimitive false
> isSynthetic false
> simple-name RuntimeDelegate
> modifier abstract,public
> annotation
> interfaces
> super-class +-java.lang.Object
> class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
> +-sun.misc.Launcher$ExtClassLoader@436a4e4b
> classLoaderHash 4fca772dAffect(row-cnt:3) cost in 112 ms. {code}
> By removing the jersey-core-1.19.3.jar file from /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/, we can see that the abstract class javax.ws.rs.ext.RuntimeDelegate is loaded by the AppClassLoader from the javax.ws.rs-api-2.1.1.jar file located in /usr/bigtop/3.2.0/usr/lib/hbase/lib/.
> At the same time, the implementation of javax.ws.rs.ext.RuntimeDelegate, which is com.sun.jersey.server.impl.provider.RuntimeDelegateImpl, is loaded from the jersey-server-1.19.jar file located in /usr/bigtop/3.2.0/usr/lib/hadoop/lib/. Note that the hadoop lib directory is appended to HBase's classpath when HBase is started, which is why it can be loaded.
> Additionally, the abstract class com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate, which inherits from RuntimeDelegate, is loaded from the jersey-core-1.19.jar file located in /usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/.
> However, even after removing the jersey-core-1.19.3.jar file, we can still see that HBase loads the same version of jersey-core-1.19.jar from the hadoop lib directory. Therefore, the root cause of the error is not a class not found due to a version conflict, but rather due to the order in which the class loader loads the classes.
>
> To verify this conclusion, the simplest way is to:
> # Before deleting the jersey-core-1.19.3.jar file from /ranger-hbase-plugin-impl/, print out the loading information of all the classes involved in the stack trace of the heap dump exception in the log.
> # After deleting the jersey-core-1.19.3.jar file, print out the loading information of all the classes involved in the stack trace of the heap dump exception in the log.
> # Use regular expressions to batch delete the timestamps from the log, and then compare the two log files to see any differences in the class loading information.
> In the Ranger 2.4 source code, Ranger implements its own class loader, RangerPluginClassLoader, which inherits from URLClassLoader. During initialization, it is passed the path to the ranger-\{componentname}{-}plugin-impl directory, and all the classes in this directory will be loaded by RangerPluginClassLoader. The parent is set to null, which in my understanding, is to block delegation loading. This is because all the classes in the ranger{-}
> {componentname}
> -plugin-impl directory are dependencies of the Ranger plugin. For example, if the version of a jar that it depends on is lower, if the delegation loading mechanism is used, it is easy to load the higher version of the class under the component/lib directory, leading to class conflict issues. Therefore, parent is set to null.
> Internally, RangerPluginClassLoader first tries to load the jar files under the ranger-\{componentname}-plugin-impl directory using the parent URLClassLoader. If it can't be loaded, it uses the componentClassLoader, which is the current thread context's class loader, to load it.
> !image-2023-04-27-16-13-44-504.png!
> Next, you can take a look at the source code of the "RangerAuthorizationCoprocessor" class in Ranger. This class initializes an instance of "rangerPluginClassLoader" and activates it. Within this class, there are pre and post hooks for various HBase operations, primarily for performing range permission-related operations. This triggers the loading of Ranger-related classes by the rangerPluginClassLoader.
> !image-2023-04-27-16-14-10-803.png!
> Why does the hook activate the Ranger class loader at the beginning and de-activate it after the hook method ends? Doesn't this approach impact performance?
> !image-2023-04-27-16-14-37-202.png!
> The reason for de-activating the Ranger class loader after the hook method ends is because the rangerPluginClassLoader code sets itself as the current thread's context class loader. If it's not deactivated, it can cause conflicts with the lower version packages loaded in the .../ranger-hbase-plugin-impl/ directory, which can cause issues during subsequent operations. Therefore, the conclusion is that this involves the rangerPluginClassLoader and the AppClassLoader of the thread's context class loader.
> We also need to understand how to load an instance of "RuntimeDelegate" and how it searches for the implementation of "javax.ws.rs.ext.RuntimeDelegate". We should also investigate why it cannot find the implementation class and instead looks for "org.glassfish.jersey.internal.RuntimeDelegateImpl", causing an error.
> First, the "RuntimeDelegate" class is loaded via the "javax.ws.rs.ext.RuntimeDelegate.findDelegate" method, which calls "FactoryFinder.find" to load the class. Once loaded, the instance must be of type "RuntimeDelegate".
> {code:java}
> private static RuntimeDelegate findDelegate() {
> try {
> Object delegate = FactoryFinder.find("javax.ws.rs.ext.RuntimeDelegate", "org.glassfish.jersey.internal.RuntimeDelegateImpl", RuntimeDelegate.class);
> if (!(delegate instanceof RuntimeDelegate)) {
> Class<RuntimeDelegate> pClass = RuntimeDelegate.class;
> String classnameAsResource = pClass.getName().replace('.', '/') + ".class";
> ClassLoader loader = pClass.getClassLoader();
> if (loader == null)
> loader = ClassLoader.getSystemClassLoader();
> URL targetTypeURL = loader.getResource(classnameAsResource);
> throw new LinkageError("ClassCastException: attempting to cast" + delegate
> .getClass().getClassLoader().getResource(classnameAsResource) + " to " + targetTypeURL);
> }
> return (RuntimeDelegate)delegate;
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> } {code}
> The process of loading the class via "FactoryFinder.find" is contained in the "javax.ws.rs.ext.FactoryFinder.find" method, as shown in the following code:
> {code:java}
> /**
> * Finds the implementation {@code Class} for the given factory name,
> * or if that fails, finds the {@code Class} for the given fallback
> * class name and create its instance. The arguments supplied MUST be
> * used in order. If using the first argument is successful, the second
> * one will not be used.
> * <p>
> * This method is package private so that this code can be shared.
> *
> * @param factoryId the name of the factory to find, which is
> * a system property.
> * @param fallbackClassName the implementation class name, which is
> * to be used only if nothing else.
> * is found; {@code null} to indicate that
> * there is no fallback class name.
> * @param service service to be found.
> * @param <T> type of the service to be found.
> * @return the instance of the specified service; may not be {@code null}.
> * @throws ClassNotFoundException if the given class could not be found
> * or could not be instantiated.
> */
> static <T> Object find(final String factoryId, final String fallbackClassName, Class<T> service) throws ClassNotFoundException {
> ClassLoader classLoader = getContextClassLoader(); try {
> Iterator<T> iterator = ServiceLoader.load(service, FactoryFinder.getContextClassLoader()).iterator(); if(iterator.hasNext()) {
> return iterator.next();
> }
> } catch (Exception | ServiceConfigurationError ex) {
> LOGGER.log(Level.FINER, "Failed to load service " + factoryId + ".", ex);
> } try {
> Iterator<T> iterator = ServiceLoader.load(service, FactoryFinder.class.getClassLoader()).iterator(); if(iterator.hasNext()) {
> return iterator.next();
> }
> } catch (Exception | ServiceConfigurationError ex) {
> LOGGER.log(Level.FINER, "Failed to load service " + factoryId + ".", ex);
> } // try to read from $java.home/lib/jaxrs.properties
> FileInputStream inputStream = null;
> String configFile = null;
> try {
> String javah = System.getProperty("java.home");
> configFile = javah + File.separator + "lib" + File.separator + "jaxrs.properties";
> File f = new File(configFile);
> if (f.exists()) {
> Properties props = new Properties();
> inputStream = new FileInputStream(f);
> props.load(inputStream);
> String factoryClassName = props.getProperty(factoryId);
> return newInstance(factoryClassName, classLoader);
> }
> } catch (Exception ex) {
> LOGGER.log(Level.FINER, "Failed to load service " + factoryId
> + " from $java.home/lib/jaxrs.properties", ex);
> } finally {
> if (inputStream != null) {
> try {
> inputStream.close();
> } catch (IOException ex) {
> LOGGER.log(Level.FINER, String.format("Error closing %s file.", configFile), ex);
> }
> }
> } // Use the system property
> try {
> String systemProp = System.getProperty(factoryId);
> if (systemProp != null) {
> return newInstance(systemProp, classLoader);
> }
> } catch (SecurityException se) {
> LOGGER.log(Level.FINER, "Failed to load service " + factoryId
> + " from a system property", se);
> } if (fallbackClassName == null) {
> throw new ClassNotFoundException(
> "Provider for " + factoryId + " cannot be found", null);
> } return newInstance(fallbackClassName, classLoader);
> }
> } {code}
> To summarize the "FactoryFinder.find" method, it uses the thread context class loader to load "javax.ws.rs.ext.RuntimeDelegate". If that fails, it attempts to load it from system properties or alternative class names. If it still cannot be found, it tries to load "org.glassfish.jersey.internal.RuntimeDelegateImpl". If it cannot be loaded, it throws an exception.
> When the /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar file was not deleted, the class loading order was as follows:
> javax.ws.rs.ext.RuntimeDelegate(appClassLoader) -> javax.ws.rs.ext.RuntimeDelegate.HeaderDelegate(RangerPluginClassLoader) -> javax.ws.rs.ext.RuntimeDelegate.HeaderDelegateProvider(RangerPluginClassLoader jersey-core-1.19.3.jar ranger-hbase-plugin-impl/jersey-core-1.19.3.jar).
> The RangerPluginClassLoader searched for the com.sun.jersey.spi.HeaderDelegateProvider implementation class in the META-INF/services/com.sun.jersey.spi.HeaderDelegateProvider file of the jersey-core-1.19.3.jar file, found the com.sun.jersey.core.impl.provider.header.LocaleProvider class in ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, and returned it.
> However, javax.ws.rs.ext.RuntimeDelegate.findDelegate would fail because delegate instanceof RuntimeDelegate returned false.
> As a fallback, it searched for org.glassfish.jersey.internal.RuntimeDelegateImpl, which was not found in the hbase classpath and caused an exception to be thrown.
> After removing ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, HBase can be started normally. The class loading order is as follows:
> javax.ws.rs.ext.RuntimeDelegate is loaded from /usr/lib/hadoop/lib/javax.ws.rs-api-2.1.1.jar. Then, the implementation class of javax.ws.rs.ext.RuntimeDelegate is searched for, which is found in /usr/lib/hadoop/lib/jersey-server-1.19.jar->META-INF/services/javax.ws.rs.ext.RuntimeDelegate->com.sun.jersey.server.impl.provider.RuntimeDelegateImpl, and successfully loaded.
> The above class loading order was determined by modifying the code in the RangerPluginClassLoader to print out the process of loading all classes.
> !image-2023-04-27-16-15-44-539.png!
> The following is a log that includes the class loading order. By analyzing this log in conjunction with the Ranger class loader code, the above results can be determined.
> [^hbase-hbase-master.err.log]
> [^hbase-hbase-master.log]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)