You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Vihang Karajgaonkar (Jira)" <ji...@apache.org> on 2021/06/22 19:25:00 UTC
[jira] [Comment Edited] (IMPALA-10759)
MetastoreServiceHandler.get_partitions_by_names_req throws
NoSuchMethodError
[ https://issues.apache.org/jira/browse/IMPALA-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367628#comment-17367628 ]
Vihang Karajgaonkar edited comment on IMPALA-10759 at 6/22/21, 7:24 PM:
------------------------------------------------------------------------
I took a brief look at this yesterday and I found that the issue happens when Impala is using 0.9.3 thrift version and hive is using 0.13.0 version. This happens because the hashCode method in the thrift generated code for HMS objects like Partition changes when you change thrift from 0.9.3 to 0.13.0. Specifically the hashCode values for the primitive fields like long now use TBaseHelper.hashCode(long) instead of the old way of add it to a ArrayList and then comparing the hashCode.
For example, in case of writeId field of the partition which is definied as a i64 in the thrift file, the hashCode is computed using TBaseHelper as seen in the diffs here https://github.com/apache/hive/commit/1945e2f67e5b09cdda40146b87e1ba492f897196#diff-505c537842790dadd6f182b07b0b216be40e050588941213220b4ae3622bd0faR877
I don't think there is a good way to "fix" this. This should get fixed automatically when Impala uses 0.11.0 thrift version. I confirmed that the build where we saw this failure did not have the Impala thrift version as 0.11.0.
was (Author: vihangk1):
I took a brief look at this yesterday and I found that the issue happens when Impala is using 0.9.3 thrift version and hive is using 0.13.0 version. This happens because the {noformat}hashCode{noformat} method in the thrift generated code for HMS objects like Partition changes when you change thrift from 0.9.3 to 0.13.0. Specifically the hashCode values for the primitive fields like long now use {noformat}TBaseHelper.hashCode{noformat} instead of the old way of add it to a ArrayList and then comparing the hashCode.
For example, in case of writeId field of the partition which is definied as a i64 in the thrift file, the hashCode is computed using TBaseHelper as seen in the diffs here https://github.com/apache/hive/commit/1945e2f67e5b09cdda40146b87e1ba492f897196#diff-505c537842790dadd6f182b07b0b216be40e050588941213220b4ae3622bd0faR877
I don't think there is a good way to "fix" this. This should get fixed automatically when Impala uses 0.11.0 thrift version. I confirmed that the build where we saw this failure did not have the Impala thrift version as 0.11.0.
> MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
> ----------------------------------------------------------------------------
>
> Key: IMPALA-10759
> URL: https://issues.apache.org/jira/browse/IMPALA-10759
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Yongzhi Chen
> Assignee: Vihang Karajgaonkar
> Priority: Critical
>
> impala-cdpd-master-core EnableCatalogdHmsCacheFlagTest.testEnableCatalogdCachingFlag test fails with following stack:
> {noformat}
> Exception in thread "pool-470-thread-1" java.lang.NoSuchMethodError: org.apache.thrift.TBaseHelper.hashCode(J)I
> at org.apache.hadoop.hive.metastore.api.Partition.hashCode(Partition.java:971)
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at org.apache.impala.catalog.CatalogHmsAPIHelper.loadAndSetFileMetadataFromFs(CatalogHmsAPIHelper.java:527)
> at org.apache.impala.catalog.metastore.MetastoreServiceHandler.get_partitions_by_names_req(MetastoreServiceHandler.java:1443)
> at org.apache.impala.catalog.metastore.CatalogMetastoreServiceHandler.get_partitions_by_names_req(CatalogMetastoreServiceHandler.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.impala.catalog.metastore.CatalogMetastoreServer$TimingInvocationHandler.invoke(CatalogMetastoreServer.java:223)
> at com.sun.proxy.$Proxy87.get_partitions_by_names_req(Unknown Source)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20087)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20066)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org