You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/05/07 22:29:04 UTC
[GitHub] [incubator-hudi] bhasudha commented on issue #836: Query hang for Presto running on JDK 11
bhasudha commented on issue #836:
URL: https://github.com/apache/incubator-hudi/issues/836#issuecomment-625529003
> just posting this in case anyone else hits this.
> i had hive (v 2.3.4) tables using 'com.uber.hoodie.hadoop.HoodieInputFormat' and COW hoodie files (written by spark 2.3.x using hoodie v 0.4.6) working with prestodb 220, java8, there was a custom hoodie jar in the presto plugin folder.
> i then tried to upgrade to prestosql 333 (which requires java11) but did not change hive inputformat definitions/version or hoodie jar in presto or how spark writing to the hoodie files. Simply trying presto (and JRE it runs on) change...
>
> First hit this error on 333:
>
> ```
> Could not initialize class com.uber.hoodie.common.table.timeline.HoodieActiveTimeline [DB Errorcode=16777221]
> "failureInfo" : {
> "type" : "io.prestosql.spi.PrestoException",
> "message" : "Could not initialize class com.uber.hoodie.common.table.timeline.HoodieActiveTimeline",
> "cause" : {
> "type" : "java.lang.NoClassDefFoundError",
> "message" : "Could not initialize class com.uber.hoodie.common.table.timeline.HoodieActiveTimeline",
> "suppressed" : [ ],
> "stack" : [ "com.uber.hoodie.common.table.HoodieTableMetaClient.getActiveTimeline(HoodieTableMetaClient.java:192)", "com.uber.hoodie.hadoop.HoodieInputFormat.listStatus(HoodieInputFormat.java:86)", "org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325)", "io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:407)", "io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:287)", "io.prestosql.plugin.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:107)", "io.prestosql.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:216)", "io.prestosql.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)", "io.prestosql.$gen.Presto_333____20200507_180703_2.run(Unknown Source)", "io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)", "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base/java.lang.Thread.run(Unknown Source)" ],
> "errorCode" : {
> "code" : 65536,
> "name" : "GENERIC_INTERNAL_ERROR",
> "type" : "INTERNAL_ERROR"
> }
> },
> "suppressed" : [ ],
> "stack" : [ "io.prestosql.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:223)", "io.prestosql.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)", "io.prestosql.$gen.Presto_333____20200507_180703_2.run(Unknown Source)", "io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)", "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base/java.lang.Thread.run(Unknown Source)" ],
> "errorCode" : {
> "code" : 16777221,
> "name" : "HIVE_UNKNOWN_ERROR",
> "type" : "EXTERNAL"
> }
> },
> ```
>
> then tried to replace the custom hoodie jar in presto plugin folder with https://repo1.maven.org/maven2/com/uber/hoodie/hoodie-presto-bundle/0.4.7/hoodie-presto-bundle-0.4.7.jar
>
> hit this error:
>
> ```
> 2020-05-07T20:01:19.405Z ERROR main io.prestosql.server.PrestoServer 'com.fasterxml.jackson.databind.ObjectMapper com.fasterxml.jackson.databind.ObjectMapper.setDefaultPropertyInclusion(com.fasterxml.jackson.annotation.JsonInclude$Value)'
> java.lang.NoSuchMethodError: 'com.fasterxml.jackson.databind.ObjectMapper com.fasterxml.jackson.databind.ObjectMapper.setDefaultPropertyInclusion(com.fasterxml.jackson.annotation.JsonInclude$Value)'
> at io.airlift.json.ObjectMapperProvider.get(ObjectMapperProvider.java:128)
> at io.airlift.json.ObjectMapperProvider.get(ObjectMapperProvider.java:47)
> at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:85)
> at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:77)
> at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:66)
> at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
> at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
> at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
> at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
> at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
> at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61)
> at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1094)
> at io.airlift.json.JsonCodecFactory.createObjectMapper(JsonCodecFactory.java:130)
> at io.airlift.json.JsonCodecFactory.jsonCodec(JsonCodecFactory.java:70)
> at io.airlift.json.JsonCodecProvider.get(JsonCodecProvider.java:43)
> at io.airlift.json.JsonCodecProvider.get(JsonCodecProvider.java:23)
> at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:85)
> at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:57)
> at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:66)
> at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
> at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
> at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
> at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
> at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
> at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47)
> at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
> at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
> at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
> at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
> at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
> at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32)
> at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
> at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
> at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
> at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
> at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
> at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
> at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
> at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
> at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
> at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
> at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
> at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
> at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
> at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
> at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
> at com.google.inject.Guice.createInjector(Guice.java:87)
> at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:262)
> at io.prestosql.plugin.hive.InternalHiveConnectorFactory.createConnector(InternalHiveConnectorFactory.java:118)
> at io.prestosql.plugin.hive.InternalHiveConnectorFactory.createConnector(InternalHiveConnectorFactory.java:78)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
> at io.prestosql.plugin.hive.HiveConnectorFactory.create(HiveConnectorFactory.java:63)
> at io.prestosql.connector.ConnectorManager.createConnector(ConnectorManager.java:349)
> at io.prestosql.connector.ConnectorManager.createCatalog(ConnectorManager.java:208)
> at io.prestosql.connector.ConnectorManager.createCatalog(ConnectorManager.java:200)
> at io.prestosql.connector.ConnectorManager.createCatalog(ConnectorManager.java:186)
> at io.prestosql.metadata.StaticCatalogStore.loadCatalog(StaticCatalogStore.java:88)
> at io.prestosql.metadata.StaticCatalogStore.loadCatalogs(StaticCatalogStore.java:68)
> at io.prestosql.server.PrestoServer.run(PrestoServer.java:118)
> at io.prestosql.$gen.Presto_333____20200507_200106_1.run(Unknown Source)
> at io.prestosql.server.PrestoServer.main(PrestoServer.java:72)
> ```
>
> then instead put https://mvnrepository.com/artifact/org.apache.hudi/hudi-presto-bundle/0.5.2-incubating in presto plugin folder and seems to work. So i guess new hudi bundle shades jackson , caters for java11 reflection and can still read the uber inputformat from hive definitions? @bhasudha
Thanks for updating this thread @tooptoop4 . I think the [pom changes]( https://github.com/apache/incubator-hudi/pull/846/files#diff-0aeecd78c8829509e3e17e8a87ddce2dR82) and [java11 support](https://github.com/apache/incubator-hudi/pull/1369) helped here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org