You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/06/04 00:39:00 UTC
[jira] [Commented] (IMPALA-11260) Catalog cache item sizes of CatalogdMetaProvider are underestimated on Java9+
[ https://issues.apache.org/jira/browse/IMPALA-11260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729030#comment-17729030 ]
ASF subversion and git services commented on IMPALA-11260:
----------------------------------------------------------
Commit 3346d070ad47b49ccbfa087e27b6ecb137b36a0a in impala's branch refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3346d070a ]
IMPALA-11260: (Addendum) Restrict add-opens to Java 9+
Restricts jvm_automatic_add_opens to only apply to Java 9+ where the
option exists. Previously it would also include it in Java 8, which
caused the JVM to ignore all options in JAVA_TOOL_OPTIONS.
Tests for Java version by running $JAVA_HOME/bin/java -version (or
"java" if JAVA_HOME is unset) and parsing version from the first line.
All JVM implementations are expected to include the version in a quoted
string, such as "1.8.0_42" and "11.0.1".
Also added add-opens flags for frontend tests.
test_no_inaccessible_objects detected this in a test run.
Testing:
- manually confirmed -agentlib options are present with both Java
8 and Java 11.
- promoted test_jvm_mem_tracking to run in all strategies, as it's fast
and ensures JAVA_TOOL_OPTIONS is honored.
Change-Id: I85953e685f6bbbd213afd93f389066e82f193ddf
Reviewed-on: http://gerrit.cloudera.org:8080/19939
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Catalog cache item sizes of CatalogdMetaProvider are underestimated on Java9+
> -----------------------------------------------------------------------------
>
> Key: IMPALA-11260
> URL: https://issues.apache.org/jira/browse/IMPALA-11260
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.1.0, Impala 3.2.0, Impala 4.0.0, Impala 3.3.0, Impala 3.4.0, Impala 3.4.1, Impala 4.1.0
> Reporter: Quanlong Huang
> Assignee: Michael Smith
> Priority: Critical
> Fix For: Impala 4.3.0
>
>
> When running local catalog mode on Java11, the Ehcache sizeof lib complains that cache sizes may be underestimated:
> {code:java}
> W0421 20:50:44.238312 9819 ObjectGraphWalker.java:251] 744e548159a57cb5:879ee74c00000000] The JVM is preventing Ehcache from accessing the subgraph beneath 'final jdk.internal.loader.URLClassPath jdk.internal.loader.ClassLoaders$AppClassLoader.ucp' - cache sizes may be underestimated as a result
> Java exception follows:
> java.lang.reflect.InaccessibleObjectException: Unable to make field final jdk.internal.loader.URLClassPath jdk.internal.loader.ClassLoaders$AppClassLoader.ucp accessible: module java.base does not "opens jdk.internal.loader" to unnamed module @6ba7383d
> at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340)
> at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280)
> at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:176)
> at java.base/java.lang.reflect.Field.setAccessible(Field.java:170)
> at org.ehcache.sizeof.ObjectGraphWalker.getAllFields(ObjectGraphWalker.java:245)
> at org.ehcache.sizeof.ObjectGraphWalker.getFilteredFields(ObjectGraphWalker.java:204)
> at org.ehcache.sizeof.ObjectGraphWalker.walk(ObjectGraphWalker.java:159)
> at org.ehcache.sizeof.SizeOf.deepSizeOf(SizeOf.java:74)
> at org.apache.impala.catalog.local.CatalogdMetaProvider$SizeOfWeigher.weigh(CatalogdMetaProvider.java:1999)
> at com.google.common.cache.LocalCache$Segment.setValue(LocalCache.java:2010)
> at com.google.common.cache.LocalCache$Segment.replace(LocalCache.java:2956)
> at com.google.common.cache.LocalCache.replace(LocalCache.java:4258)
> at org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:540)
> at org.apache.impala.catalog.local.CatalogdMetaProvider.loadIcebergApiTable(CatalogdMetaProvider.java:1056)
> at org.apache.impala.catalog.local.LocalIcebergTable.loadIcebergTableViaMetaProvider(LocalIcebergTable.java:87)
> at org.apache.impala.catalog.local.LocalTable.load(LocalTable.java:107)
> at org.apache.impala.catalog.local.LocalDb.getTable(LocalDb.java:127)
> at org.apache.impala.analysis.StmtMetadataLoader.getMissingTables(StmtMetadataLoader.java:310)
> at org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:165)
> at org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:141)
> at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2014)
> at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1926)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1750)
> at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164){code}
> Similar errors on other classes:
> {code}
> The JVM is preventing Ehcache from accessing the subgraph beneath 'final jdk.internal.loader.AbstractClassLoaderValue jdk.internal.loader.AbstractClassLoaderValue$Sub.this$0' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'final jdk.internal.loader.URLClassPath jdk.internal.loader.ClassLoaders$AppClassLoader.ucp' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.lang.Object jdk.internal.loader.AbstractClassLoaderValue$Sub.key' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.lang.String java.lang.module.Configuration.targetPlatform' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.lang.String java.lang.module.ModuleDescriptor.mainClass' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.lang.String java.lang.module.ModuleDescriptor.name' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.lang.String java.lang.module.ModuleDescriptor.rawVersionString' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.lang.module.ModuleDescriptor$Version java.lang.module.ModuleDescriptor.version' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.List java.lang.module.Configuration.parents' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Map java.lang.module.Configuration.graph' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Map java.lang.module.Configuration.nameToModule' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Map jdk.internal.loader.BuiltinClassLoader.moduleToReader' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Map jdk.internal.loader.BuiltinClassLoader.nameToModule' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Map jdk.internal.module.ServicesCatalog.map' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.Configuration.modules' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.exports' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.modifiers' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.opens' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.packages' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.provides' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.requires' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.uses' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final jdk.internal.loader.BuiltinClassLoader jdk.internal.loader.BuiltinClassLoader.parent' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final jdk.internal.loader.URLClassPath jdk.internal.loader.BuiltinClassLoader.ucp' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private jdk.internal.reflect.ConstructorAccessorImpl jdk.internal.reflect.DelegatingConstructorAccessorImpl.delegate' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private volatile java.lang.ref.SoftReference jdk.internal.loader.BuiltinClassLoader.resourceCache' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private volatile java.util.List java.lang.module.Configuration.allConfigurations' - cache sizes may be underestimated as a result
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org