Review Request 74426: RANGER-4228: Kafka2.8 can't restart after enable ranger plugin

[jialiang cai <ji...@gmail.com>]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74426/
-----------------------------------------------------------

Review request for ranger, Don Bosco Durai, Brijesh Bhalala, Dhaval Rajpara, Madhan Neethiraj, Dharmesh Makwana, Nitin Galave, and Selvamohan Neethiraj.


Bugs: RANGER-4228
    https://issues.apache.org/jira/browse/RANGER-4228


Repository: ranger


Description
-------

env:
hadoop 3.3.4
kafka 2.8
ranegr 2.4
os: centos7 amd64

Kafka2.8 can't restart after enable ranger plugin

1.Missing hadoop shade package

A temporary solution is to copy the required JAR files from Hadoop


cp /usr/bigtop/current/hadoop-hdfs-client/lib/hadoop-shaded-guava-1.1.1.jar /usr/bigtop/current/kafka-broker/libs/

[2023-05-06 14:33:22,254] ERROR [KafkaServer id=1001] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.NoClassDefFoundError: org/apache/hadoop/thirdparty/com/google/common/base/Preconditions
at org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:430)
at org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:443)
at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:525)
at org.apache.ranger.plugin.service.RangerBasePlugin.<init>(RangerBasePlugin.java:79)
at org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:100)
at org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:95)
at kafka.security.authorizer.AuthorizerWrapper.configure(AuthorizerWrapper.scala:86)
at kafka.server.KafkaServer.$anonfun$startup$11(KafkaServer.scala:352)
at kafka.server.KafkaServer.$anonfun$startup$11$adapted(KafkaServer.scala:352)
at scala.Option.foreach(Option.scala:407)
at kafka.server.KafkaServer.startup(KafkaServer.scala:352)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.com.google.common.base.Preconditions
at java.lang.ClassLoader.findClass(ClassLoader.java:530)
at org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:316)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:160)
... 13 more 

https://issues.apache.org/jira/browse/HADOOP-14284

Starting from Hadoop 3, Hadoop uses the shaded Guava library located in the hadoop-shade jar instead of the original Guava library. However, unlike HBase or Hive, Kafka does not add the Hadoop lib to its class path during startup. Therefore, in a big data cluster using Hadoop 3, the HBase and Hive plugins can start without errors, but Kafka may report errors due to the lack of the hadoop-shade jar in its class path. As a temporary solution, the necessary jar files can be copied from Hadoop to Kafka.


2.Missing common-compress package

A temporary solution is to copy the required JAR files from Hadoop."

[2023-05-08 10:15:18,025] ERROR [KafkaServer id=1001] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.NoClassDefFoundError: org/apache/commons/compress/archivers/tar/TarArchiveInputStream
at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNonNativeIO(RawLocalFileSystem.java:864)
at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:855)
at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:816)
at org.apache.hadoop.security.alias.KeyStoreProvider.stashOriginalFilePermissions(KeyStoreProvider.java:78)
at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:324)
at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:86)
at org.apache.hadoop.security.alias.KeyStoreProvider.<init>(KeyStoreProvider.java:49)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:42)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:35)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68)
at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:91)
at org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialProviders(RangerCredentialProvider.java:69)
at org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialString(RangerCredentialProvider.java:47)
at org.apache.ranger.audit.provider.MiscUtil.getCredentialString(MiscUtil.java:455)
at org.apache.ranger.audit.destination.SolrAuditDestination.getKeyManagers(SolrAuditDestination.java:348)
at org.apache.ranger.audit.destination.SolrAuditDestination.connect(SolrAuditDestination.java:131)
at org.apache.ranger.audit.destination.SolrAuditDestination.init(SolrAuditDestination.java:96)
at org.apache.ranger.audit.provider.AuditProviderFactory.init(AuditProviderFactory.java:183)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:234)
at org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:105)
at org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:95)
at kafka.security.authorizer.AuthorizerWrapper.configure(AuthorizerWrapper.scala:86)
at kafka.server.KafkaServer.$anonfun$startup$11(KafkaServer.scala:352)
at kafka.server.KafkaServer.$anonfun$startup$11$adapted(KafkaServer.scala:352)
at scala.Option.foreach(Option.scala:407)
at kafka.server.KafkaServer.startup(KafkaServer.scala:352)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.compress.archivers.tar.TarArchiveInputStream
at java.lang.ClassLoader.findClass(ClassLoader.java:530)
at org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:316)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:160)
... 28 more
[2023-05-08 10:15:18,028] INFO [KafkaServer id=1001] shutting down (kafka.server.KafkaServer) 

cp /usr/bigtop/current/hadoop-hdfs-client/lib/commons-compress-1.21.jar /usr/bigtop/current/kafka-broker/libs

https://issues.apache.org/jira/browse/RANGER-4228


Diffs
-----

  distro/src/main/assembly/plugin-kafka.xml ed8ef6159 


Diff: https://reviews.apache.org/r/74426/diff/1/


Testing
-------

after apply this patch ,kafka start smoonthly


Thanks,

jialiang cai