You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Mithun Radhakrishnan (JIRA)" <ji...@apache.org> on 2015/03/19 07:28:38 UTC
[jira] [Commented] (HIVE-9845) HCatSplit repeats information making
input split data size huge
[ https://issues.apache.org/jira/browse/HIVE-9845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368588#comment-14368588 ]
Mithun Radhakrishnan commented on HIVE-9845:
--------------------------------------------
There are several problems:
1. {{PartInfo}} stores and serializes the full {{TableInfo}} for every {{HCatSplit}} instance, even though that information is immutable;
2. {{PartInfo}} stores class-names for StorageHandler, SerDe, InputFormat and OutputFormat. It's likely that a lot of this is identical to the Table's info. I've changed the serialization not to include it if it doesn't differ from the Table.
3. Every {{HCatSplit}} stores the table's column schema separately, in spite of having this information available both in the {{InputJobInfo}} in the configuration, and in the {{TableInfo}} within the {{PartInfo}}. Again, redundant and wasteful.
I've changed the above. My testing with a Pig-script on a wide table (75 columns) with a query spanning 45 partitions and 1000 splits looks promising:
{code}
# Before:
-rw-r--r-- 10 mithunr users 55335466 2015-03-19 05:56 /user/mithunr/.staging/job_1426414782073_303408/job.split
# After:
-rw-r--r-- 10 mithunr users 2643046 2015-03-19 06:01 /user/mithunr/.staging/job_1426414782073_303697/job.split
{code}
Will post patch after cleaning it up a tad.
> HCatSplit repeats information making input split data size huge
> ---------------------------------------------------------------
>
> Key: HIVE-9845
> URL: https://issues.apache.org/jira/browse/HIVE-9845
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
> Reporter: Rohini Palaniswamy
> Assignee: Mithun Radhakrishnan
>
> Pig on Tez jobs with larger tables hit PIG-4443. Running on HDFS data which has even triple the number of splits(100K+ splits and tasks) does not hit that issue.
> {code}
> HCatBaseInputFormat.java:
> //Call getSplit on the InputFormat, create an
> //HCatSplit for each underlying split
> //NumSplits is 0 for our purposes
> org.apache.hadoop.mapred.InputSplit[] baseSplits =
> inputFormat.getSplits(jobConf, 0);
> for(org.apache.hadoop.mapred.InputSplit split : baseSplits) {
> splits.add(new HCatSplit(
> partitionInfo,
> split,allCols));
> }
> {code}
> Each hcatSplit duplicates partition schema and table schema.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)