[jira] [Commented] (HIVE-9845) HCatSplit repeats information making input split data size huge

["Hive QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-9845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389772#comment-14389772 ] 

Hive QA commented on HIVE-9845:
-------------------------------



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12708487/HIVE-9845.3.patch

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 8692 tests executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-smb_mapjoin_8.q - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3226/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3226/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3226/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12708487 - PreCommit-HIVE-TRUNK-Build

> HCatSplit repeats information making input split data size huge
> ---------------------------------------------------------------
>
>                 Key: HIVE-9845
>                 URL: https://issues.apache.org/jira/browse/HIVE-9845
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>            Reporter: Rohini Palaniswamy
>            Assignee: Mithun Radhakrishnan
>         Attachments: HIVE-9845.1.patch, HIVE-9845.3.patch
>
>
> Pig on Tez jobs with larger tables hit PIG-4443. Running on HDFS data which has even triple the number of splits(100K+ splits and tasks) does not hit that issue.
> {code}
> HCatBaseInputFormat.java:
>  //Call getSplit on the InputFormat, create an
>       //HCatSplit for each underlying split
>       //NumSplits is 0 for our purposes
>       org.apache.hadoop.mapred.InputSplit[] baseSplits = 
>         inputFormat.getSplits(jobConf, 0);
>       for(org.apache.hadoop.mapred.InputSplit split : baseSplits) {
>         splits.add(new HCatSplit(
>             partitionInfo,
>             split,allCols));
>       }
> {code}
> Each hcatSplit duplicates partition schema and table schema.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)