Hive Query Error: Cannot obtain block length

[Arun Patel <ar...@gmail.com>]

I am trying to do log analytics on the logs created by Flume.  Hive queries
are failing with below error.  "hadoop fs -cat" command works on all these
open files. Is there a way to read these open files?   My requirement is to
read the data from open files too.  I am using tez as execution engine.

select b.ts as ts1, a.ts as ts2, a.tid from log_v_rst1 a join log_v_rst2 b
on a.tid = b.tid ;

Error: Error while processing statement: FAILED: Execution Error, return
code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
vertexName=Map 2, vertexId=vertex_1467055804180_0003_2_00,
diagnostics=[Task failed, taskId=task_1467055804180_0003_2_00_000006,
diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
task:java.lang.RuntimeException: java.lang.RuntimeException:
java.io.IOException: java.io.IOException: Cannot obtain block length for
LocatedBlock{BP-854133642-XX.XXX.XX.XX-1460753641159:blk_1073771231_33308;
getBlockSize()=0; corrupt=false; offset=0;
locs=[DatanodeInfoWithStorage[XX.XXX.XX.XXX:1019,DS-3712a177-3199-4b61-bd34-598d04edc6d9,DISK]]}
        at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
        at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
        at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
        at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
        at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
        at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
        at
org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)