You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Bunakov Ilya (Jira)" <ji...@apache.org> on 2025/06/09 15:16:00 UTC
[jira] [Updated] (BIGTOP-4438) Problem compatibility of Hadoop components (HDFS, Mapreduce, Hive, Spark)
[ https://issues.apache.org/jira/browse/BIGTOP-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bunakov Ilya updated BIGTOP-4438:
---------------------------------
Description:
In the new version of BigTop 3.4.0 there are problems with compatibility of Hadoop components (HDFS, Mapreduce, Hive).
When running Mapreduce applications there are problems with dependencies, example:
~ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-3.3.6.jar teragen 100000000 /tmp/teragenout
java.lang.RuntimeException: java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter overrides final method getTimeStamp.()J
at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:149)
at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:181)
at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:266)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:347)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:344)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:344)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:471)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:448)
at org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:434)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:478)
Caused by: java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter overrides final method getTimeStamp.()J
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.log4j.builders.BuilderManager.<clinit>(BuilderManager.java:58)
at org.apache.log4j.config.Log4j1Configuration.<init>(Log4j1Configuration.java:46)
at org.apache.log4j.config.PropertiesConfiguration.<init>(PropertiesConfiguration.java:90)
at org.apache.log4j.config.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:58)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:499)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:404)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:318)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:690)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:46)
at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:41)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.fs.FileSystem.<clinit>(FileSystem.java:186)
at org.apache.hadoop.fs.local.RawLocalFs.<init>(RawLocalFs.java:57)
at org.apache.hadoop.fs.local.RawLocalFs.<init>(RawLocalFs.java:43)
at org.apache.hadoop.fs.local.LocalFs.<init>(LocalFs.java:37)
at org.apache.hadoop.fs.local.LocalFs.<init>(LocalFs.java:51)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:143)
... 12 more
Similar stack trace when using the tez on yarn:
~ beeline
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:635)
Connected to: Apache Hive (version 4.0.1)
Driver: Hive JDBC (version 4.0.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 4.0.1 by Apache Hive
0: jdbc:hive2://>
Failing this attempt.Diagnostics: [2025-06-09 18:13:46.513]Application application_1749479392838_0007 initialization failed (exitCode=255) with output: main : command provided 0
java.lang.RuntimeException: java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter overrides final method getTimeStamp.()J
at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:149)
at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:181)
at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:266)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:347)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:344)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:344)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:471)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:448)
at org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:434)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:478)
Caused by: java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter overrides final method getTimeStamp.()J
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.log4j.builders.BuilderManager.<clinit>(BuilderManager.java:58)
at org.apache.log4j.config.Log4j1Configuration.<init>(Log4j1Configuration.java:46)
at org.apache.log4j.config.PropertiesConfiguration.<init>(PropertiesConfiguration.java:90)
at org.apache.log4j.config.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:58)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:499)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:404)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:318)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:690)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:46)
at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:41)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.fs.FileSystem.<clinit>(FileSystem.java:186)
at org.apache.hadoop.fs.local.RawLocalFs.<init>(RawLocalFs.java:57)
at org.apache.hadoop.fs.local.RawLocalFs.<init>(RawLocalFs.java:43)
at org.apache.hadoop.fs.local.LocalFs.<init>(LocalFs.java:37)
at org.apache.hadoop.fs.local.LocalFs.<init>(LocalFs.java:51)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:143)
... 12 more
Hey did anyone manage to get Hive 4.0.1 working with Spark 3.5.3? SparkSQL can query {{show databases}} and successfully displays all available databases, but invoking {{select * from xyz}} fails with {{{}HiveException: unable to fetch table xyz. Invalid method name 'get_table'{}}}. Adding the jars from hive to spark and specifying {{spark.sql.hive.metastore.version 4.0.1}} throws an error about unsupported version and all queries fail. Is there a workaround?
was:
In the new version of BigTop 3.4.0 there are problems with compatibility of Hadoop components (HDFS, Mapreduce, Hive).
When running Mapreduce applications there are problems with dependencies, example:
~ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-3.3.6.jar teragen 100000000 /tmp/teragenout
java.lang.RuntimeException: java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter overrides final method getTimeStamp.()J
at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:149)
at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:181)
at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:266)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:347)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:344)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:344)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:471)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:448)
at org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:434)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:478)
Caused by: java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter overrides final method getTimeStamp.()J
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.log4j.builders.BuilderManager.<clinit>(BuilderManager.java:58)
at org.apache.log4j.config.Log4j1Configuration.<init>(Log4j1Configuration.java:46)
at org.apache.log4j.config.PropertiesConfiguration.<init>(PropertiesConfiguration.java:90)
at org.apache.log4j.config.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:58)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:499)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:404)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:318)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:690)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:46)
at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:41)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.fs.FileSystem.<clinit>(FileSystem.java:186)
at org.apache.hadoop.fs.local.RawLocalFs.<init>(RawLocalFs.java:57)
at org.apache.hadoop.fs.local.RawLocalFs.<init>(RawLocalFs.java:43)
at org.apache.hadoop.fs.local.LocalFs.<init>(LocalFs.java:37)
at org.apache.hadoop.fs.local.LocalFs.<init>(LocalFs.java:51)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:143)
... 12 more
Similar stack trace when using the tez on yarn.
Hey did anyone manage to get Hive 4.0.1 working with Spark 3.5.3? SparkSQL can query {{show databases}} and successfully displays all available databases, but invoking {{select * from xyz}} fails with {{{}HiveException: unable to fetch table xyz. Invalid method name 'get_table'{}}}. Adding the jars from hive to spark and specifying {{spark.sql.hive.metastore.version 4.0.1}} throws an error about unsupported version and all queries fail. Is there a workaround?
> Problem compatibility of Hadoop components (HDFS, Mapreduce, Hive, Spark)
> -------------------------------------------------------------------------
>
> Key: BIGTOP-4438
> URL: https://issues.apache.org/jira/browse/BIGTOP-4438
> Project: Bigtop
> Issue Type: Bug
> Components: hadoop, hive
> Affects Versions: 3.4.0
> Environment: bigtop 3.4.0
> ubuntu-24.04
> hadoop 3.3.6
> hive 4.0.1
> Reporter: Bunakov Ilya
> Priority: Blocker
>
> In the new version of BigTop 3.4.0 there are problems with compatibility of Hadoop components (HDFS, Mapreduce, Hive).
> When running Mapreduce applications there are problems with dependencies, example:
> ~ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-3.3.6.jar teragen 100000000 /tmp/teragenout
> java.lang.RuntimeException: java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter overrides final method getTimeStamp.()J
> at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:149)
> at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:181)
> at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:266)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:347)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:344)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:344)
> at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:471)
> at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:448)
> at org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:434)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:478)
> Caused by: java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter overrides final method getTimeStamp.()J
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at org.apache.log4j.builders.BuilderManager.<clinit>(BuilderManager.java:58)
> at org.apache.log4j.config.Log4j1Configuration.<init>(Log4j1Configuration.java:46)
> at org.apache.log4j.config.PropertiesConfiguration.<init>(PropertiesConfiguration.java:90)
> at org.apache.log4j.config.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:58)
> at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:499)
> at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:404)
> at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:318)
> at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:690)
> at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
> at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
> at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
> at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
> at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
> at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
> at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
> at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
> at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:46)
> at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:41)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
> at org.apache.hadoop.fs.FileSystem.<clinit>(FileSystem.java:186)
> at org.apache.hadoop.fs.local.RawLocalFs.<init>(RawLocalFs.java:57)
> at org.apache.hadoop.fs.local.RawLocalFs.<init>(RawLocalFs.java:43)
> at org.apache.hadoop.fs.local.LocalFs.<init>(LocalFs.java:37)
> at org.apache.hadoop.fs.local.LocalFs.<init>(LocalFs.java:51)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:143)
> ... 12 more
> Similar stack trace when using the tez on yarn:
> ~ beeline
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:635)
> Connected to: Apache Hive (version 4.0.1)
> Driver: Hive JDBC (version 4.0.1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 4.0.1 by Apache Hive
> 0: jdbc:hive2://>
> Failing this attempt.Diagnostics: [2025-06-09 18:13:46.513]Application application_1749479392838_0007 initialization failed (exitCode=255) with output: main : command provided 0
> java.lang.RuntimeException: java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter overrides final method getTimeStamp.()J
> at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:149)
> at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:181)
> at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:266)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:347)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:344)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:344)
> at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:471)
> at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:448)
> at org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:434)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:478)
> Caused by: java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter overrides final method getTimeStamp.()J
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at org.apache.log4j.builders.BuilderManager.<clinit>(BuilderManager.java:58)
> at org.apache.log4j.config.Log4j1Configuration.<init>(Log4j1Configuration.java:46)
> at org.apache.log4j.config.PropertiesConfiguration.<init>(PropertiesConfiguration.java:90)
> at org.apache.log4j.config.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:58)
> at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:499)
> at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:404)
> at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:318)
> at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:690)
> at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
> at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
> at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
> at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
> at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
> at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
> at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
> at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
> at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:46)
> at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:41)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
> at org.apache.hadoop.fs.FileSystem.<clinit>(FileSystem.java:186)
> at org.apache.hadoop.fs.local.RawLocalFs.<init>(RawLocalFs.java:57)
> at org.apache.hadoop.fs.local.RawLocalFs.<init>(RawLocalFs.java:43)
> at org.apache.hadoop.fs.local.LocalFs.<init>(LocalFs.java:37)
> at org.apache.hadoop.fs.local.LocalFs.<init>(LocalFs.java:51)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:143)
> ... 12 more
> Hey did anyone manage to get Hive 4.0.1 working with Spark 3.5.3? SparkSQL can query {{show databases}} and successfully displays all available databases, but invoking {{select * from xyz}} fails with {{{}HiveException: unable to fetch table xyz. Invalid method name 'get_table'{}}}. Adding the jars from hive to spark and specifying {{spark.sql.hive.metastore.version 4.0.1}} throws an error about unsupported version and all queries fail. Is there a workaround?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)