[jira] [Created] (TEZ-3403) Empty partition issue with Hive on TEZ

["Ashish Kumar (JIRA)" <ji...@apache.org>]

Ashish Kumar created TEZ-3403:
---------------------------------

             Summary: Empty partition issue with Hive on TEZ
                 Key: TEZ-3403
                 URL: https://issues.apache.org/jira/browse/TEZ-3403
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Ashish Kumar


Hi,

I'm experiencing few failures with TEZ regarding Hive partitions. Even though there is no partition column used in the query still it is giving partition file path not found error.

I'm trying to run below query with Hive on TEZ and getting some partition issue. The same query is working fine with MR engine. Used table is external one and having partitions on year and month columns. I've seen few times 

*Query:*

*select count(crn) as bookings, month(to_date(from_utc_timestamp(pickup_date,'IST'))) as month from bookings_table and year=2016 group by month(to_date(from_utc_timestamp(pickup_date,'IST')));*

*Error:*

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.io.IOException: While processing file s3n://dataplatform-prod-store/warehouse/ola_bookings/year=2016/month=1. null 
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:78) 
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:292) 
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:148) 
... 14 more 
Caused by: java.io.IOException: java.io.IOException: While processing file s3n://<bucket>/warehouse/bookings_table/year=2016/month=1. null 
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) 
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) 
at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:372) 
at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79) 
at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33) 
at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:118) 
at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:137) 
at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:113) 
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) 
... 16 more 

*Another error for other query:*

DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:4 
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1470240409111_2339_1_06, diagnostics=[Vertex vertex_1470240409111_2339_1_06 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: app_sessions initializer failed, vertex=vertex_1470240409111_2339_1_06 [Map 1], java.io.FileNotFoundException: No such file or directory: s3n://dataplatform-prod-store/warehouse/dp_appSessions/year=2015/month=02/day=14/hour=03 
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1078) 
at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:783) 
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1500) 
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1540) 
at org.apache.hadoop.fs.FileSystem$4.(FileSystem.java:1704) 
at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1703) 
at org.apache.hadoop.mapred.InputPathProcessor.perPathComputation(InputPathProcessor.java:235) 
at org.apache.hadoop.mapred.InputPathProcessor.access$000(InputPathProcessor.java:28) 
at org.apache.hadoop.mapred.InputPathProcessor$2.run(InputPathProcessor.java:338) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)