You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Syed Shameerur Rahman (Jira)" <ji...@apache.org> on 2021/01/13 08:44:00 UTC
[jira] [Comment Edited] (HIVE-24584) IndexOutOfBoundsException from
Kryo when running msck repair
[ https://issues.apache.org/jira/browse/HIVE-24584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263983#comment-17263983 ]
Syed Shameerur Rahman edited comment on HIVE-24584 at 1/13/21, 8:43 AM:
------------------------------------------------------------------------
[~amagyar] - As per my understanding all msck command flow are defaulted to use MsckPartitionExpressionProxy unless EXPRESSION_PROXY_CLASS is explicitly given.
So is there any reason for explicitly setting EXPRESSION_PROXY_CLASS or am i missing anything? Because the above mentioned problem doesn't arise if you use the default path.
{code:java}
public static Configuration getMsckConf(Configuration conf) {
// the only reason we are using new conf here is to override EXPRESSION_PROXY_CLASS
Configuration metastoreConf = MetastoreConf.newMetastoreConf(new Configuration(conf));
metastoreConf.set(MetastoreConf.ConfVars.EXPRESSION_PROXY_CLASS.getVarname(),
metastoreConf.get(MetastoreConf.ConfVars.EXPRESSION_PROXY_CLASS.getVarname(),
MsckPartitionExpressionProxy.class.getCanonicalName()));
return metastoreConf;
}
{code}
was (Author: srahman):
[~amagyar] - As per my understanding all msck command flow are defaulted to use MsckPartitionExpressionProxy unless EXPRESSION_PROXY_CLASS is explicitly given.
So is there any reason for explicitly setting EXPRESSION_PROXY_CLASS or am i missing anything? Because the above mentioned problem doesn't arise if you use the default path.
> IndexOutOfBoundsException from Kryo when running msck repair
> ------------------------------------------------------------
>
> Key: HIVE-24584
> URL: https://issues.apache.org/jira/browse/HIVE-24584
> Project: Hive
> Issue Type: Bug
> Reporter: Attila Magyar
> Assignee: Attila Magyar
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The following exception is coming when running "msck repair table t1 sync partitions".
> {code:java}
> java.lang.IndexOutOfBoundsException: Index: 97, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:657) ~[?:1.8.0_232]
> at java.util.ArrayList.get(ArrayList.java:433) ~[?:1.8.0_232]
> at org.apache.hive.com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:60) ~[hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:834) ~[hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:684) ~[hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:211) ~[hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectFromKryo(SerializationUtilities.java:814) ~[hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeExpressionFromKryo(SerializationUtilities.java:775) ~[hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.deserializeExpr(PartitionExpressionForMetastore.java:116) [hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.filterPartitionsByExpr(PartitionExpressionForMetastore.java:88) [hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT] {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)