You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Aravindan Vijayan (Jira)" <ji...@apache.org> on 2020/01/22 00:13:00 UTC
[jira] [Comment Edited] (HDDS-2914) Certain Hive queries started to
fail on generating splits
[ https://issues.apache.org/jira/browse/HDDS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020673#comment-17020673 ]
Aravindan Vijayan edited comment on HDDS-2914 at 1/22/20 12:12 AM:
-------------------------------------------------------------------
Based on looking at the way Hive OrcInputFormat uses the block locations, there seems to be a misunderstanding in the way the block offset was handled in HDDS-2188.
The following is the method used by Hive to create a map of offset -> locations
{code}
public TreeMap<Long, BlockLocation> getLocationsWithOffset(FileSystem fs,
FileStatus status) throws IOException {
TreeMap<Long, BlockLocation> offsetBlockMap = new TreeMap<Long, BlockLocation>();
BlockLocation[] locations = getLocations(fs, status);
for (BlockLocation location : locations) {
offsetBlockMap.put(location.getOffset(), location);
}
return offsetBlockMap;
}
{code}
It looks like the offset field in BlockLocation refers to the "offset of the block in the set of blocks that are part of the file" rather than the offset of the block in the physical file that contains the block. In our case, the latter is always 0, and when we use 0 in the Block Locations, the above code overwrites the same entry in the Map. I am planning on testing this theory out on a cluster where the hive queries are failing.
[~msingh] Please let me know if this makes sense.
was (Author: avijayan):
Based on looking at the way org.apache.hadoop.hive.ql.io.orc.OrcInputFormat uses the block locations, there seems to be a misunderstanding in the way the block offset was handled in HDDS-2188.
The following is the method used by Hive to create a map of offset -> locations
{code}
public TreeMap<Long, BlockLocation> getLocationsWithOffset(FileSystem fs,
FileStatus status) throws IOException {
TreeMap<Long, BlockLocation> offsetBlockMap = new TreeMap<Long, BlockLocation>();
BlockLocation[] locations = getLocations(fs, status);
for (BlockLocation location : locations) {
offsetBlockMap.put(location.getOffset(), location);
}
return offsetBlockMap;
}
{code}
It looks like the offset field in BlockLocation refers to the "offset of the block in the set of blocks that are part of the file" rather than the offset of the block in the physical file that contains the block. In our case, the latter is always 0, and when we use 0 in the Block Locations, the above code overwrites the same entry in the Map. I am planning on testing this theory out on a cluster where the hive queries are failing.
[~msingh] Please let me know if this makes sense.
> Certain Hive queries started to fail on generating splits
> ---------------------------------------------------------
>
> Key: HDDS-2914
> URL: https://issues.apache.org/jira/browse/HDDS-2914
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Istvan Fajth
> Assignee: Aravindan Vijayan
> Priority: Critical
>
> After updating a cluster where I am running TPCDS queries, some queries started to fail.
> The update happened from an early dec state to the jan 10 state of master.
> Most likely the addition of HDDS-2188 is related to the problem, but it is still under investigation.
> The exception I see in the queries:
> {code}
> [ERROR] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex Input: inventory initializer failed, vertex=vertex_XXXX [Map 1]
> org.apache.tez.dag.app.dag.impl.AMUserCodeException: java.lang.RuntimeException: ORC split generation failed with exception: java.io.IOException: File o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 should have had overlap on block starting at 0
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:328)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
> at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
> at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
> at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
> at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
> at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:110)
> at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: ORC split generation failed with exception: java.io.IOException: File o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 should have had overlap on block starting at 0
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1915)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:532)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:789)
> at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> ... 5 more
> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: File o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 should have had overlap on block starting at 0
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1909)
> ... 17 more
> Caused by: java.io.IOException: File o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 should have had overlap on block starting at 0
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.createSplit(OrcInputFormat.java:1506)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.generateOrUpdateSplit(OrcInputFormat.java:1678)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.generateSplitsFromStripes(OrcInputFormat.java:1661)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1609)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565)
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {code}
> I haven't found any more specific stack traces so far.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org