You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Rajat Khandelwal <ra...@gmail.com> on 2015/07/30 14:48:14 UTC
Review Request 36939: HIVE-11376: CombineHiveInputFormat is falling
back to
HiveInputFormat in case codecs are found for one of the input files
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36939/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-11376
https://issues.apache.org/jira/browse/HIVE-11376
Repository: hive-git
Description
-------
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L379
This is the exact code snippet:
{noformat}
/ Since there is no easy way of knowing whether MAPREDUCE-1597 is present in the tree or not,
// we use a configuration variable for the same
if (this.mrwork != null && !this.mrwork.getHadoopSupportsSplittable()) {
// The following code should be removed, once
// https://issues.apache.org/jira/browse/MAPREDUCE-1597 is fixed.
// Hadoop does not handle non-splittable files correctly for CombineFileInputFormat,
// so don't use CombineFileInputFormat for non-splittable files
//ie, dont't combine if inputformat is a TextInputFormat and has compression turned on
{noformat}
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 33b67dd7b0fde41f81f8d86ea8c83d29c631e3d7
ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 1de7e4073f5eea4c7be8423a7ebe6a89cb51d9f1
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 693d8c7e9f956999b2da33593d780e37ddf2b3b8
ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 3217df27bb5731a1dcd5db1ae17c5bdff2e3fbfc
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 76926e79729d9ea4823de0ffc9b1e5bac6364842
Diff: https://reviews.apache.org/r/36939/diff/
Testing
-------
Thanks,
Rajat Khandelwal
Re: Review Request 36939: HIVE-11376: CombineHiveInputFormat is
falling back
to HiveInputFormat in case codecs are found for one of the input files
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36939/#review93869
-----------------------------------------------------------
Ship it!
Ship It!
- Amareshwari Sriramadasu
On July 30, 2015, 1:19 p.m., Rajat Khandelwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36939/
> -----------------------------------------------------------
>
> (Updated July 30, 2015, 1:19 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-11376
> https://issues.apache.org/jira/browse/HIVE-11376
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L379
>
> This is the exact code snippet:
>
> {noformat}
>
> / Since there is no easy way of knowing whether MAPREDUCE-1597 is present in the tree or not,
> // we use a configuration variable for the same
> if (this.mrwork != null && !this.mrwork.getHadoopSupportsSplittable()) {
> // The following code should be removed, once
> // https://issues.apache.org/jira/browse/MAPREDUCE-1597 is fixed.
> // Hadoop does not handle non-splittable files correctly for CombineFileInputFormat,
> // so don't use CombineFileInputFormat for non-splittable files
>
> //ie, dont't combine if inputformat is a TextInputFormat and has compression turned on
>
> {noformat}
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 33b67dd7b0fde41f81f8d86ea8c83d29c631e3d7
> ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 1de7e4073f5eea4c7be8423a7ebe6a89cb51d9f1
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 693d8c7e9f956999b2da33593d780e37ddf2b3b8
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 3217df27bb5731a1dcd5db1ae17c5bdff2e3fbfc
> ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 76926e79729d9ea4823de0ffc9b1e5bac6364842
>
> Diff: https://reviews.apache.org/r/36939/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Rajat Khandelwal
>
>
Re: Review Request 36939: HIVE-11376: CombineHiveInputFormat is
falling back
to HiveInputFormat in case codecs are found for one of the input files
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36939/
-----------------------------------------------------------
(Updated July 30, 2015, 6:49 p.m.)
Review request for hive.
Bugs: HIVE-11376
https://issues.apache.org/jira/browse/HIVE-11376
Repository: hive-git
Description
-------
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L379
This is the exact code snippet:
{noformat}
/ Since there is no easy way of knowing whether MAPREDUCE-1597 is present in the tree or not,
// we use a configuration variable for the same
if (this.mrwork != null && !this.mrwork.getHadoopSupportsSplittable()) {
// The following code should be removed, once
// https://issues.apache.org/jira/browse/MAPREDUCE-1597 is fixed.
// Hadoop does not handle non-splittable files correctly for CombineFileInputFormat,
// so don't use CombineFileInputFormat for non-splittable files
//ie, dont't combine if inputformat is a TextInputFormat and has compression turned on
{noformat}
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 33b67dd7b0fde41f81f8d86ea8c83d29c631e3d7
ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 1de7e4073f5eea4c7be8423a7ebe6a89cb51d9f1
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 693d8c7e9f956999b2da33593d780e37ddf2b3b8
ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 3217df27bb5731a1dcd5db1ae17c5bdff2e3fbfc
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 76926e79729d9ea4823de0ffc9b1e5bac6364842
Diff: https://reviews.apache.org/r/36939/diff/
Testing
-------
Thanks,
Rajat Khandelwal