You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Aman Sinha (Jira)" <ji...@apache.org> on 2021/05/18 15:41:00 UTC
[jira] [Commented] (IMPALA-10057) TransactionKeepalive
NoClassDefFoundError floods logs during JDBC_TEST/FE_TEST
[ https://issues.apache.org/jira/browse/IMPALA-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17347009#comment-17347009 ]
Aman Sinha commented on IMPALA-10057:
-------------------------------------
Saw the same TransactionKeepalive NoClassDefFoundError in a recent impala-asf-master-core-s3-data-cache build which ended up creating huge catalogd and impalad ERROR logs. Details in IMPALA-10707.
> TransactionKeepalive NoClassDefFoundError floods logs during JDBC_TEST/FE_TEST
> ------------------------------------------------------------------------------
>
> Key: IMPALA-10057
> URL: https://issues.apache.org/jira/browse/IMPALA-10057
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.0
> Reporter: Joe McDonnell
> Priority: Major
> Labels: flaky
>
> For the both the normal tests and the docker-based tests, the Impala logs generated during the FE_TEST/JDBC_TEST can be huge:
>
> {noformat}
> $ du -c -h fe_test/ee_tests
> 4.0K fe_test/ee_tests/minidumps/statestored
> 4.0K fe_test/ee_tests/minidumps/impalad
> 4.0K fe_test/ee_tests/minidumps/catalogd
> 16K fe_test/ee_tests/minidumps
> 352K fe_test/ee_tests/profiles
> 81G fe_test/ee_tests
> 81G total{noformat}
> Creating a tarball of these logs takes 10 minutes. The Impalad/catalogd logs are filled with this error over and over:
> {noformat}
> E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception thrown
> Java exception follows:
> java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
> at org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError: org/apache/impala/common/TransactionKeepalive$HeartbeatContext
> ... 2 more
> Caused by: java.lang.ClassNotFoundException: org.apache.impala.common.TransactionKeepalive$HeartbeatContext
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 2 more{noformat}
> Two interesting points:
> # The frontend/jdbc tests are passing, so all of these errors in the impalad logs are not impacting tests.
> # These errors aren't concurrently with any of the other tests (ee tests, custom cluster tests, etc).
> This is happening on normal core runs (including the GVO job that does FE_TEST/JDBC_TEST) on both Ubuntu and Centos 7. It is also happening on docker-based tests. A theory is that FE_TEST/JDBC_TEST have an Impala cluster running and then invoke maven to run the tests. Maven could manipulate jars while Impala is running. Maybe there is a race-condition or conflict when manipulating those jars that could cause the NoClassDefFoundError. It makes no sense for Impala not to be able to find TransactionKeepalive$HeartbeatContext.
> When it happens, it is in a tight loop, printing the message more than once per millisecond. It fills the ERROR, WARNING, and INFO logs with that message, sometimes for multiple Impalads and/or catalogd.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org