[jira] [Created] (HIVE-18572) The record readers; InputFormat needs to be fixed for Tez as it generates 1 split

["gurmukh singh (JIRA)" <ji...@apache.org>]

gurmukh singh created HIVE-18572:
------------------------------------

             Summary: The record readers; InputFormat needs to be fixed for Tez as it generates 1 split
                 Key: HIVE-18572
                 URL: https://issues.apache.org/jira/browse/HIVE-18572
             Project: Hive
          Issue Type: Bug
    Affects Versions: 2.1.0
            Reporter: gurmukh singh


The record reader needs to be fixed for tez, as it generates only 1 split due to the {color:#333333}MRv2 CombineInputFormat broke that rule{color}.

This has been fixed in MR but not Tez.

I am seeing a strange behaviour in tez; it is seeing all data as a single split under hive, where as MR see all 79 files. This is causing all the data to go to a single map

TEZ Processing
INFO  : Partition trusted.usage\{ds=20180126, periode=1200} stats: [numFiles=1, numRows=79575067, totalSize=3.164.605.993, rawDataSize=112439569671]
ELAPSED TIME: 1958.99 s

MR Processing
Partition trusted.usage\{ds=20180126, periode=1200} stats: [numFiles=79, numRows=79575067, totalSize=3172280778, rawDataSize=112418416260]
ELAPSED TIME: 65 s

Log Tez
2018-01-29 16:50:04,825 [INFO] [InputInitializer \{Map 1} #0] |split.TezMapredSplitsGrouper|: Desired splits: 381 too large.  Desired splitLength: 8311476 Min splitLength: 50331648 New desired splits: 381 Final desired splits: 381 All splits have localhost: false Total length: 19166265870 Original splits: 1
2018-01-29 16:50:04,825 [INFO] [InputInitializer \{Map 1} #0] |split.TezMapredSplitsGrouper|: Using original number of splits: 1 desired splits: 381
2018-01-29 16:50:04,826 [INFO] [InputInitializer \{Map 1} #0] |tez.SplitGrouper|: Original split size is 1 grouped split size is 1, for bucket: 1
2018-01-29 16:50:04,827 [INFO] [InputInitializer \{Map 1} #0] |tez.HiveSplitGenerator|: Number of grouped splits: 1
2018-01-29 16:50:04,846 [INFO] [InputInitializer \{Map 1} #0] |dag.RootInputInitializerManager|: Succeeded InputInitializer for Input: usage on vertex vertex_1517207496169_0085_1_00 [Map 1]
2018-01-29 16:50:04,848 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Cannot init vertex: vertex_1517207496169_0085_1_00 [Map 1] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2018-01-29 16:50:04,848 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Got updated RootInputsSpecs: \{usage=forAllWorkUnits=true, update=[1]}
2018-01-29 16:50:04,859 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Vertex vertex_1517207496169_0085_1_00 [Map 1] parallelism set to 1

As per discussion with Gopal Vijayaraghavan:

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java#L494
 that line, right there MRv2 CombineInputFormat broke that rule, so the record readers had to be fixed to handle it https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java#L312



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)