You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Mithun Radhakrishnan (JIRA)" <ji...@apache.org> on 2015/03/19 07:28:38 UTC

[jira] [Commented] (HIVE-9845) HCatSplit repeats information making input split data size huge

    [ https://issues.apache.org/jira/browse/HIVE-9845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368588#comment-14368588 ] 

Mithun Radhakrishnan commented on HIVE-9845:
--------------------------------------------

There are several problems:

1. {{PartInfo}} stores and serializes the full {{TableInfo}} for every {{HCatSplit}} instance, even though that information is immutable;
2. {{PartInfo}} stores class-names for StorageHandler, SerDe, InputFormat and OutputFormat. It's likely that a lot of this is identical to the Table's info. I've changed the serialization not to include it if it doesn't differ from the Table.
3. Every {{HCatSplit}} stores the table's column schema separately, in spite of having this information available both in the {{InputJobInfo}} in the configuration, and in the {{TableInfo}} within the {{PartInfo}}. Again, redundant and wasteful.

I've changed the above. My testing with a Pig-script on a wide table (75 columns) with a query spanning 45 partitions and 1000 splits looks promising:

{code}
# Before:
-rw-r--r--  10 mithunr users   55335466 2015-03-19 05:56 /user/mithunr/.staging/job_1426414782073_303408/job.split
# After:
-rw-r--r--  10 mithunr users    2643046 2015-03-19 06:01 /user/mithunr/.staging/job_1426414782073_303697/job.split
{code}

Will post patch after cleaning it up a tad.

> HCatSplit repeats information making input split data size huge
> ---------------------------------------------------------------
>
>                 Key: HIVE-9845
>                 URL: https://issues.apache.org/jira/browse/HIVE-9845
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>            Reporter: Rohini Palaniswamy
>            Assignee: Mithun Radhakrishnan
>
> Pig on Tez jobs with larger tables hit PIG-4443. Running on HDFS data which has even triple the number of splits(100K+ splits and tasks) does not hit that issue.
> {code}
> HCatBaseInputFormat.java:
>  //Call getSplit on the InputFormat, create an
>       //HCatSplit for each underlying split
>       //NumSplits is 0 for our purposes
>       org.apache.hadoop.mapred.InputSplit[] baseSplits = 
>         inputFormat.getSplits(jobConf, 0);
>       for(org.apache.hadoop.mapred.InputSplit split : baseSplits) {
>         splits.add(new HCatSplit(
>             partitionInfo,
>             split,allCols));
>       }
> {code}
> Each hcatSplit duplicates partition schema and table schema.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)