You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/02/11 21:58:00 UTC
[jira] [Updated] (HDDS-2914) Certain Hive queries started to fail
on generating splits
[ https://issues.apache.org/jira/browse/HDDS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDDS-2914:
---------------------------------
Labels: pull-request-available (was: )
> Certain Hive queries started to fail on generating splits
> ---------------------------------------------------------
>
> Key: HDDS-2914
> URL: https://issues.apache.org/jira/browse/HDDS-2914
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Istvan Fajth
> Assignee: Aravindan Vijayan
> Priority: Critical
> Labels: pull-request-available
>
> After updating a cluster where I am running TPCDS queries, some queries started to fail.
> The update happened from an early dec state to the jan 10 state of master.
> Most likely the addition of HDDS-2188 is related to the problem, but it is still under investigation.
> The exception I see in the queries:
> {code}
> [ERROR] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex Input: inventory initializer failed, vertex=vertex_XXXX [Map 1]
> org.apache.tez.dag.app.dag.impl.AMUserCodeException: java.lang.RuntimeException: ORC split generation failed with exception: java.io.IOException: File o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 should have had overlap on block starting at 0
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:328)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
> at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
> at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
> at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
> at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
> at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:110)
> at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: ORC split generation failed with exception: java.io.IOException: File o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 should have had overlap on block starting at 0
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1915)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:532)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:789)
> at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> ... 5 more
> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: File o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 should have had overlap on block starting at 0
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1909)
> ... 17 more
> Caused by: java.io.IOException: File o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 should have had overlap on block starting at 0
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.createSplit(OrcInputFormat.java:1506)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.generateOrUpdateSplit(OrcInputFormat.java:1678)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.generateSplitsFromStripes(OrcInputFormat.java:1661)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1609)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {code}
> I haven't found any more specific stack traces so far.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org