[jira] [Commented] (IMPALA-11260) Catalog cache item sizes of CatalogdMetaProvider are underestimated on Java9+

["Michael Smith (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-11260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719565#comment-17719565 ] 

Michael Smith commented on IMPALA-11260:
----------------------------------------

The CatalogdMetaProvider stores lots of different objects, including Java standard library containers. Implementing size calculation without reflection looks challenging, and approximating size would probably not be sufficient.

There are alternative implementations, but none of them are compelling due to similar problems with Java 9+ or incompatible licenses.

I think our best bet for the moment is to work around this. At some point we may still need to fork ehcache if it continues to be effectively unmaintained; at some point in the future https://openjdk.org/jeps/8249196 may also become an option.

> Catalog cache item sizes of CatalogdMetaProvider are underestimated on Java9+
> -----------------------------------------------------------------------------
>
>                 Key: IMPALA-11260
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11260
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Catalog
>    Affects Versions: Impala 3.1.0, Impala 3.2.0, Impala 4.0.0, Impala 3.3.0, Impala 3.4.0, Impala 3.4.1, Impala 4.1.0
>            Reporter: Quanlong Huang
>            Assignee: Quanlong Huang
>            Priority: Critical
>
> When running local catalog mode on Java11, the Ehcache sizeof lib complains that cache sizes may be underestimated:
> {code:java}
> W0421 20:50:44.238312  9819 ObjectGraphWalker.java:251] 744e548159a57cb5:879ee74c00000000] The JVM is preventing Ehcache from accessing the subgraph beneath 'final jdk.internal.loader.URLClassPath jdk.internal.loader.ClassLoaders$AppClassLoader.ucp' - cache sizes may be underestimated as a result
> Java exception follows:
> java.lang.reflect.InaccessibleObjectException: Unable to make field final jdk.internal.loader.URLClassPath jdk.internal.loader.ClassLoaders$AppClassLoader.ucp accessible: module java.base does not "opens jdk.internal.loader" to unnamed module @6ba7383d
>         at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340)
>         at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280)
>         at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:176)
>         at java.base/java.lang.reflect.Field.setAccessible(Field.java:170)
>         at org.ehcache.sizeof.ObjectGraphWalker.getAllFields(ObjectGraphWalker.java:245)
>         at org.ehcache.sizeof.ObjectGraphWalker.getFilteredFields(ObjectGraphWalker.java:204)
>         at org.ehcache.sizeof.ObjectGraphWalker.walk(ObjectGraphWalker.java:159)
>         at org.ehcache.sizeof.SizeOf.deepSizeOf(SizeOf.java:74)
>         at org.apache.impala.catalog.local.CatalogdMetaProvider$SizeOfWeigher.weigh(CatalogdMetaProvider.java:1999)
>         at com.google.common.cache.LocalCache$Segment.setValue(LocalCache.java:2010)
>         at com.google.common.cache.LocalCache$Segment.replace(LocalCache.java:2956)
>         at com.google.common.cache.LocalCache.replace(LocalCache.java:4258)
>         at org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:540)
>         at org.apache.impala.catalog.local.CatalogdMetaProvider.loadIcebergApiTable(CatalogdMetaProvider.java:1056)
>         at org.apache.impala.catalog.local.LocalIcebergTable.loadIcebergTableViaMetaProvider(LocalIcebergTable.java:87)
>         at org.apache.impala.catalog.local.LocalTable.load(LocalTable.java:107)
>         at org.apache.impala.catalog.local.LocalDb.getTable(LocalDb.java:127)
>         at org.apache.impala.analysis.StmtMetadataLoader.getMissingTables(StmtMetadataLoader.java:310)
>         at org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:165)
>         at org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:141)
>         at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2014)
>         at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1926)
>         at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1750)
>         at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164){code}
> Similar errors on other classes:
> {code}
> The JVM is preventing Ehcache from accessing the subgraph beneath 'final jdk.internal.loader.AbstractClassLoaderValue jdk.internal.loader.AbstractClassLoaderValue$Sub.this$0' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'final jdk.internal.loader.URLClassPath jdk.internal.loader.ClassLoaders$AppClassLoader.ucp' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.lang.Object jdk.internal.loader.AbstractClassLoaderValue$Sub.key' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.lang.String java.lang.module.Configuration.targetPlatform' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.lang.String java.lang.module.ModuleDescriptor.mainClass' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.lang.String java.lang.module.ModuleDescriptor.name' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.lang.String java.lang.module.ModuleDescriptor.rawVersionString' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.lang.module.ModuleDescriptor$Version java.lang.module.ModuleDescriptor.version' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.List java.lang.module.Configuration.parents' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Map java.lang.module.Configuration.graph' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Map java.lang.module.Configuration.nameToModule' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Map jdk.internal.loader.BuiltinClassLoader.moduleToReader' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Map jdk.internal.loader.BuiltinClassLoader.nameToModule' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Map jdk.internal.module.ServicesCatalog.map' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.Configuration.modules' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.exports' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.modifiers' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.opens' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.packages' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.provides' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.requires' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final java.util.Set java.lang.module.ModuleDescriptor.uses' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final jdk.internal.loader.BuiltinClassLoader jdk.internal.loader.BuiltinClassLoader.parent' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private final jdk.internal.loader.URLClassPath jdk.internal.loader.BuiltinClassLoader.ucp' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private jdk.internal.reflect.ConstructorAccessorImpl jdk.internal.reflect.DelegatingConstructorAccessorImpl.delegate' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private volatile java.lang.ref.SoftReference jdk.internal.loader.BuiltinClassLoader.resourceCache' - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private volatile java.util.List java.lang.module.Configuration.allConfigurations' - cache sizes may be underestimated as a result
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org