You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2022/09/13 14:50:00 UTC
[jira] [Commented] (IMPALA-11528) hive-exec.pom doesn't include UDAF class
[ https://issues.apache.org/jira/browse/IMPALA-11528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603610#comment-17603610 ]
ASF subversion and git services commented on IMPALA-11528:
----------------------------------------------------------
Commit 4e813b7085c995a7244ef886b00c22e9d93cc80c in impala's branch refs/heads/master from Steve Carlin
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=4e813b708 ]
IMPALA-11528: Catalogd should start up with a corrupt Hive function.
This commit handles the case for a specific kind of corrupt function
within the Hive Metastore in the following situation:
A valid Hive SQL function gets created in HMS. This UDF is written in
Java and must derive from the "UDF" class. After creating this function
in Impala, we then replace the underlying jar file with a class that
does NOT derive from the "UDF" class.
In this scenario, catalogd should reject the function and still start
up gracefully. Before this commit, catalogd wasn't coming up. The
reason for this was because the Hive function
FunctionUtils.getUDFClassType() has a dependency on UDAF and was
throwing a LinkageError exception, so we need to include the UDAF
class in the shaded jar.
Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Reviewed-on: http://gerrit.cloudera.org:8080/18927
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Tamas Mate <tm...@apache.org>
> hive-exec.pom doesn't include UDAF class
> ----------------------------------------
>
> Key: IMPALA-11528
> URL: https://issues.apache.org/jira/browse/IMPALA-11528
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 4.1.0
> Reporter: Gabor Kaszab
> Priority: Major
>
> For hive-exec we load "*UDF*" that excludes the UDAF class:
> https://github.com/apache/impala/blob/master/java/shaded-deps/hive-exec/pom.xml#L102
> As a result when a UDAF is being loaded to catalog we get a NoClassDefFoundException.
> {code:java}
> I0819 09:20:07.777845 1 HiveUdfLoader.java:63] Loading UDF 'eu.radoop.datahandler.hive.udf.GenericUDAFCorrelationMatrix' from file:/tmp/e5a348f5-753a-485a-b37d-2a1420b09df7.jar
> I0819 09:20:07.780457 1 MetastoreEventsProcessor.java:700] Metastore event processing restarted. Last synced event id was updated from 902310 to 902310
> I0819 09:20:07.780704 1 jni-util.cc:286] java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/exec/UDAF
> at org.apache.hadoop.hive.ql.exec.FunctionUtils.getUDFClassType(FunctionUtils.java:157)
> at org.apache.impala.hive.executor.HiveUdfLoader.<init>(HiveUdfLoader.java:68)
> at org.apache.impala.hive.executor.HiveUdfLoader.createWithLocalPath(HiveUdfLoader.java:155)
> at org.apache.impala.hive.executor.HiveJavaFunctionFactoryImpl.create(HiveJavaFunctionFactoryImpl.java:47)
> at org.apache.impala.hive.executor.HiveJavaFunctionFactoryImpl.create(HiveJavaFunctionFactoryImpl.java:67)
> at org.apache.impala.catalog.CatalogServiceCatalog.loadJavaFunctions(CatalogServiceCatalog.java:1756)
> at org.apache.impala.catalog.CatalogServiceCatalog.invalidateDb(CatalogServiceCatalog.java:1862)
> at org.apache.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:1994)
> at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:166)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.exec.UDAF
> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 9 more
> I0819 09:20:07.780738 1 status.cc:129] NoClassDefFoundError: org/apache/hadoop/hive/ql/exec/UDAF
> CAUSED BY: ClassNotFoundException: org.apache.hadoop.hive.ql.exec.UDAF
> @ 0xd99193
> @ 0x162f6d9
> @ 0xd5f447
> @ 0xd2ba46
> @ 0xc85b28
> @ 0xbd6fd0
> @ 0x7ff4f9bf7554
> @ 0xc7ba86
> E0819 09:20:07.781023 1 catalog.cc:87] NoClassDefFoundError: org/apache/hadoop/hive/ql/exec/UDAF
> CAUSED BY: ClassNotFoundException: org.apache.hadoop.hive.ql.exec.UDAF
> . Impalad exiting.
> {code}
> In Impla 3.4 we handled this exception gracefully but apaprently in 4.1 Catalog gets terminated by this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org