You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2024/01/23 08:20:00 UTC
[jira] [Commented] (IMPALA-12743) Incremental stats are filtered out by HMS due to HIVE-27114
[ https://issues.apache.org/jira/browse/IMPALA-12743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809799#comment-17809799 ]
Quanlong Huang commented on IMPALA-12743:
-----------------------------------------
This causes FE test failures in CatalogTest.testPullIncrementalStats
{noformat}
java.lang.AssertionError: expected:<11> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at org.apache.impala.catalog.CatalogTest.expectStatistics(CatalogTest.java:702)
at org.apache.impala.catalog.CatalogTest.testPullIncrementalStats(CatalogTest.java:734)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495){noformat}
Uploaded a patch for review: https://gerrit.cloudera.org/c/20937/
> Incremental stats are filtered out by HMS due to HIVE-27114
> -----------------------------------------------------------
>
> Key: IMPALA-12743
> URL: https://issues.apache.org/jira/browse/IMPALA-12743
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> HIVE-27114 adds a new property, "hive.metastore.partitions.parameters.exclude.pattern", which defaults to "impala_intermediate_stats_chunk%". This excludes the incremental stats in the parameters returned to HMS client.
> Impala is the only client that consumes this parameter. We should set it to an empty string. Otherwise, incremental stats can't be loaded when deploying Impala on newer versions of Hive. The error is
> {noformat}
> W0122 20:30:53.508314 17348 HdfsPartition.java:1444] Failed to set partition stats for table functional.alltypesagg partition year=2010/month=1/day=1
> Java exception follows:
> org.apache.impala.common.ImpalaRuntimeException: Missing stats chunk: 0
> at org.apache.impala.catalog.PartitionStatsUtil.partStatsBytesFromParameters(PartitionStatsUtil.java:108)
> at org.apache.impala.catalog.HdfsPartition$Builder.extractAndCompressPartStats(HdfsPartition.java:1441)
> at org.apache.impala.catalog.HdfsPartition$Builder.setMsPartition(HdfsPartition.java:1417)
> at org.apache.impala.catalog.HdfsTable.createOrUpdatePartitionBuilder(HdfsTable.java:959)
> at org.apache.impala.catalog.HdfsTable.createPartitionBuilder(HdfsTable.java:948)
> at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:729)
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1299)
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1192)
> at org.apache.impala.catalog.TableLoader.load(TableLoader.java:144)
> at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:248)
> at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:245)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org